Software Engineering Hiring and Firing

The jump in interest rates to the highest level in over 20 years that hit in summer 2023 for the US, UK and many other countries is still impacting the Software industry. Rates may be due to drop soon, but currently it has choked off investment, upped borrowing costs and lead to many software companies making engineers redundant to please the markets. 

For the UK the estimate is around 8% of software industry jobs made redundant. Although strangely, the overall trend in vacancies for software engineers continues to march upwards, the initial surge after the pandemic dipped last summer but has now recovered.
But if you work in the industry you are bound to have colleagues and friends who have been made redundant, if you are lucky enough to have not been impacted personally.

Given recent history, I thought it may be worth reflecting on my personal experience of the whole hiring and firing process, in the tech industry. It is a UK centric view, but the companies I have worked for in the last 8 years are US software companies.

I have been fired, hired and conducted technical interviews to hire others. Giving me a few different perspectives. 

This post is NOT about getting your first Software job

I first got a coding job in the public sector and it was as a self taught web developer in the 1990s, before web development was a thing you could get a degree in. So I initially got a  job in IT support, volunteered to act up (ie no pay increase) and built some websites that were needed, then became a full time web developer through a portfolio of work, ie sites. 

Today junior developers may have to prove themselves suitable by artificial measures. I skipped these, so I do not have any professional certifications, or any to recommend. I also don't know how to ace coding algorithm or personality profile assessments.

Once you are 5-10 years in to a software career - none of those approaches are used for hiring decisions. 

Only large companies are likely to subject you to them, and that is really out of fairness on the juniors who have to go through them, and to screen out dodgy applicants. Screening just needs to be passed, it will not have any input into whether you get the job. Hence Acing the coding interview as promoted by sites such as LeetCode is not even a thing, only passing coding exercise systems in order to start, or switch to, a career as a developer. I would recommend starting an open source project instead, to demonstrate you can actually code.

The majority of small to medium software companies and of job vacancies require experience and in effect have no vacancies for the most junior software grades with less than 3 years under their belt. So they tend not to use any of these filtering methods. They just want to see proof that you are already a developer, and usually base that on face to face interviews and examples of your code you provide them. So much like how I was originally hired back in the 1990s.

I have only been subject to a LeetCode style test once, which was for a generic job application, ie hiring for numbers of SREs of various seniority, for a FANNG.

 F I R E D 

When you get that unexpected one to one Zoom call with you manager appear in your calendar these days, it is unlikely to be great news 😓

In the majority of cases the firing process, or to be more polite, redundancy, is all about balancing the finances of the whole company or institution. As such it is very unlikely to be about you.

Of course people are also fired as individuals for various reasons, one of which is actually not being any good at their job, failing to get along with their manager, being a bad culture fit, jobs turning out not to be what was advertised, or expressing political views. Since unlike where I once worked, in the UK Education sector, where 50% of staff are union members, US software companies will have less than 1% membership, so don't tend to respond well to dissent.

Mostly this happens via failing probation, at around 15%  then maybe another 5% annually for disciplinary / performance improvement failure.

If you want to try getting individually fired then go the overemployed route. Get two or three jobs at once and test how long it takes before the company notices you giving 110% is now only 40% and fire you. The rule of thumb is the larger the company, the longer it takes!

But this post's focus isn't about individual firing, its about organizational hiring and firing.

Firing Reasons

1. A company may be doing badly in a slow long term way, so it has to chop as part of a restructure and downsize to attempt to fix that.
2. Alternatively the company could be doing really well. So it gets the attention of a big investment company and is bought up and merged with its rival. To fix overlap and justify the merger - both companies lose 20% of staff.
3. Maybe it needs to pivot towards a new area (currently likely to be AI) and so chop 20% of its staff so it can hire 15% experienced, and pricey, AI developers.
4. Or it may just have had a one off external impacting event that hit it financially. So to balance the earnings for that year and keep its share price good, it chops a bunch of staff. It will rehire next year, when it suits the balance sheet. This is the example in which I was made redundant along with 5% of staff, it was a big company, so that was a few thousand people globally.
5. Finally it may be an industry wide phenomenon as it is with the current redundancies in the software industry. A world clamp down on easy cheap loans means investment company driven industries such as tech. are no longer awash with spare cash. Cut backs look good right now, and keep the share price high.
   Hence redundancies that are nothing to do with the industry itself or its future prospects.

That is mirrored in who is fired. Companies do not keep a log book of gold stars and black marks against each employee. They do not use organizational triggered rounds of redundancies to select individuals to fire. They certainly have not got the capability to accurately determine all the best employees and only fire the worst ones. You will be fired based on what part of the organization you are in, how much it is valued in the current strategy and how much you cost vs others who could do your job. If you are currently between teams / or in a new team or role which has yet to establish itself, when the music stops - like musical chairs, bad luck you are out.

The only personal element may be if a whole team is seen as under performing or difficult to manage it might be axed. No matter that it contains a star performer. Decisions may also be geographic. Lets axe the Greek office, save by withdrawing engineering from a country, which is again how I was made redundant, the rest of my team was in Greece.
Alternatively it may be, fire under 20 staff from each country, to avoid more burdensome regulation for bulk layoffs.

The organization could create an insecure / downturn atmosphere to encourage staff to leave. Because its a lot cheaper for people to leave than the company paying out redundancy settlements.

Redundancy keeps the average employees 😐

As a result in response to significant redundancies an organisation will tend to lose more of the best employees - since they are the most able to move, the most likely to get a big pay rise if they move and the least likely to want to stick around if they see negative organisational change.  The software industry has very high staff turnover at almost 20%. Out weighing any nominal idea of removing less efficient staff. 

If a company handles things well it may only lose a representative productivity range of staff from best to worst. But a bulk redundancy process is likely to lead to the biggest loss in the top talent, get rid of slightly more of the bottom dwellers and so result in maximising the mediocre!

In summary the answer to 'Why me?' in group redundancies is "because you were there" ... and you didn't have a personal friendship with the CEO 😉. Of course that is why new CEOs are often brought in to restructure - the first step of which is to take an axe to the current C-suite. 

Some of the best software engineers I have worked with have been made redundant at some point in their career. Group redundancies are not about you or how well you do your job. But taking it personally and challenging the messenger, with why me?, as demonstrated by recent viral videos, is an understandable emotional response to rejection, and the misguided belief that work aims to be some form of meritocracy, in the same way college might.

LIFO and FIFO

LIFO rather than FIFO is the norm in Firing. New hires are less likely to have established themselves as essential to the company, and have less personal connections within it. More importantly many countries redundancy legislation doesn't kick in until over 2 years of employment and the longer you have been employed the more the company will have to pay to terminate you.

Which means a new hire who has uprooted for their new tech job, will be the most likely to find themselves losing that job when bulk redundancies hit.
But FIFO has its place, next would be older engineers. Some companies don't even hire hands on engineers much over the age of 40, anyway. But staff near retirement have at most only a few years left to contribute and may cost more for the same grade. So encouraging early retirement can be part of the bulk redundancy process.

Prejudicial Firing

Whilst redundancy is all about costs and not about your personal performance. That is not to say companies who pass the redundancy choices down to junior managers may not end up with firing disproportionate numbers of workers who are not from the same background as their manager, ie white USA males, ideally younger than the manager. But prejudice is not personal either. That is pretty much what defines it as prejudice, a pre-judgement of people based on physical characteristics rather than their ability at the job. Also people are least likely to fire staff that they have the most in common with, resulting in prejudicial firing. 
Unfortunately it seems many companies with a good diversity policy for hiring, may not have adequate ones for firing. Again resulting in losing more of the higher performing staff.

I have heard of a case where someone got a new manager, who on joining was told to cut from his team, so he fired everyone outside the USA. The worker was so keen to stay at their current employer they went over the head of their manager to senior management and asked for their redundancy to be repealed. Since they had been at the company many years and personally knew senior management, this worked.
Alternatively a more purely cost based restructure may hire all developers from cheaper countries and fire most of them in the US. As happened with Google's Python team recently.

Fight for your job?

The company may set up a pool process for bulk redundancies if numbers are high enough per country, where you can fight for a place on the lifeboat of remaining positions. 

In both cases I would recommend that you don't waste time on a company that doesn't value you. If you do stay you risk, dealing with the bulk redundancy aftermath. Which will be present unless the redundancies were for a pivot (3) or one off event (4).
An increased workload, pay freezes, no bonus, needing to over work to justify being kept on, plus a negative work atmosphere.

In a case where I stayed after the department I was in was axed, I had to reapply for a new job which was moved to a different division. The work was less worthwhile and at the time, the employment of in-house software developers as a whole, was questioned as being unnecessary for the organisation. I outstayed my welcome for 18 months of legacy commercial software support, before getting the message and quitting.

Lesson learnt, if you must ask to to stay in your company, via senior management, a pool or reapplication. Make sure you look around and apply for other jobs outside of it at the same time.

You also miss out on a minimum of a couple of months tax free pay as a settlement.

On the other hand, if the redundancy round is for a more minor pivot, and you are happy in the role, it may be well worth staying around to see how things pan out.

Of course you may get no choice in the matter, in which case, get straight into GET HIRED mode, and start the job search. If you can manage it fast enough, you will benefit financially from the whole process. Although if the reason is (5) a sector wide reduction, then it will be take longer and be harder to obtain the usual 20% pay increase that a new position can offer.

 H I R E D 

Why change jobs (aside from being fired!)

• It is a lot easier to get a pay rise or promotion by changing companies, than being promoted internally. To fast track your career to a principal or architect top IC role. Or just get a pay rise.
• Changing jobs gives you much wider experience, of different technology, approaches and cultures. Making you a better engineer.
• If you have been in your current job over 10 years without significant internal promotions or changes of role then it is detrimental to your CV, indicating you are stuck in a rut and unable to handle change, eg. new technology.
• You want to shift sectors.
  I changed from public sector web developer, to commercial cloud engineer with one move.
• You want to get into new technology that is not used in your current role.
  I changed from a Python, Ruby config management automation engineer to a Kubernetes Golang engineer with another.
• You want to change your role in tech, or leave it entirely. For example get out of sales as a solution architect and back into a more technical role as an SRE.

On that basis many software engineers change jobs every 2 or 3 years for part of their careers. Its expected, the average engineer in a FAANG stays less than 3 years.

Of course you probably need to be in a job for at least 2 years to fully master it.
If your CV has loads of similar positions where you barely make it past the probation period, its marking you out as a failure at those roles == Fail hiring at the first step, the HR CV check.

Upskilling

The other problem is that changing jobs to change roles, even if its just to use a new language or framework can be blocked by roles requiring experience in that area on the CV to get interviewed for the job in the first place. For software engineering that is less of an issue. Since tech changes faster than any other sector.
You just need to prove you have a range of experience and software languages and are willing to learn, early in a technology boom. To catch the cloud engineer bus, I got a job in it in 2016. The US cloud sector was $8 billion back then. It is $600 billion now. Similarly to get on board with Golang and Kubernetes in 2019. In the first few years of a tech boom most companies will initially have to cross train engineers without direct experience. The corollary of that is that in the current downturn attempting to pivot to an established technology, which k8s has become, is going to be much harder.

Market rates

Clearly ML ops and AI data science are current booming areas. The demand so far outstrips current supply that for switching to a more junior Python AI role in them may pay as well as a senior Django web developer for example.

So around £60k for a junior role, but in 3-4 years it should jump to at least £100k for a senior AI engineer. Of course for US salaries add 30%, plus usually free medical, life insurance etc. The lower tax rates cancels out the higher cost of living in the US ... so its UK salary +30% in real terms*. Researching the going rate for the particular role, technical skills and sector you are applying for is a necessary part of the hiring process. In order that you don't let recruitment bargain you down too low.

* Note that geographic software pay differences are why you often come across engineers of other nationalities emigrating to, and working in the higher paying countries, USA, Canada and Australia. I have worked with many people from the UK and Europe who live in the USA, and Indians who live there or Europe, for example.
Of course as a cheap foreign worker myself, I too stick with US companies partly because they pay rather more than UK ones, even if a lot lower than what I would get if I moved there 😉

Now is the time when such a switch will be easier to accomplish without having to work nights doing courses, certifications and personal projects. The usual means of demonstrating your ability without any work experience.

The caveat here is that moving jobs in a down turn, as we are arguably experiencing currently, can depress the market salary rates and if you are already at the top of those when made redundant, can mean you have to take a pay cut for a year or two rather than face the cost of long term unemployment.

The hiring process for an experienced software engineer role

Interview to Offer should take a month.

If not then the recruitment is likely for a group of roles in an expansion process and from screening and CV, you are not one of the top candidates. You may waiting on the backlog of potential interviewees for a couple of extra months before it properly kicks off.
Or you are told the post is no longer available, sorry!
Even if you would eventually get a post it may stretch your redundancy settlement. Therefore I would not bother pursuing any application process that is looking to be stretching on past 6 weeks.
Start date will be 5 weeks from contract (partly to cater for notice, referee and compliance checks etc)

That makes it 2 months minimum from applying for a role to starting.

The process will consist of a technical assessment task and at least 3 interviews, screening, manager and technical.
With another for introduction to team mates / office etc. which is unlikely to have any effect on the hiring decision unless you and your potential new manager take an instant personal dislike to each other.

The HR screening interview, just checks you are a genuine candidate for the job.

The Manager interview similarly is more about checking you will fit in with the company and team, plus that you have basic personal communication skills.

The Technical Interview is what matters

Passing the technical interview is what really decides whether you will get a job offer. Sometimes the tech interview may be split into two, one more task and questionnaire based and the other more discussion. Often the initial task part will be given as WFH.

The technical interview will consist of technical questions to explore whether you have the knowledge and experience required, plus some thing to confirm you can write code and discuss that code, for a developer or SRE role. For the former it would likely be application code whilst for the latter automation code.
For a more purely system administration / IT support role it will involve specifying your processes for resolving issues.

If you are unlucky and it is an in person interview, you may have to whiteboard pseudo code live in response to a changing task described to you on the spot. Although I have only had that once. More common, especially for hybrid / remote roles, is the take away task. To be completed in a 'few hours' at most.

It is possible that either of the above could be replaced by another source for your code. Talking through one of your open source packages, if you have any. Or talking through one or two longer automated coding exercise assessed tasks. I have never come across either of these though.

The main point is that the core of any technical interview for a developer related role will involve talking through code you have written, as a kicking off point to check your understanding of the code, how it could be improved, how you would tackle scaling, or a new exemplar functional requirement. Its faults and features.

You will be asked to talk through past code or technical work in a more generic manner in response to standard questions along the lines of examples of your past work that show how you fit the job. 

Preparing for the Technical Interview

It doesn't take much to work out that a 20% pay rise is worth, a day's worth of work a week.
Assuming you stay in your new job for 2 or 3 years - that is equivalent to 6 months pay.
On that basis even doing a week of work to apply to, and prepare for a single job is still very well worth it, if you get the job.

Adopting a scatter gun approach, ie applying with a generic CV and covering letter to 10 or more jobs, is a waste of time in my view. If you need a new job, then it should be one you are genuinely interested in and research. That means probably you should only have a maximum of 3 tailored applications on the go at once. Even when I was made redundant (and about to get married) I think I limited myself to 4 job applications in total, with one primary one that thankfully I did end up getting.

There are many sites that can advise how best to do that, based on the framework that your hiring will be decided upon I have outlined above. I think preparing some Challenge Action Result stories targeted at the details of the new employer is useful. Plus spending a day or so refining that '2 hours'  development task. Researching the company and preparing specific questions and perhaps suggestions for your interviewers.

Being a Technical Interviewer

From the other side of the table, clearly candidates need to show sufficient competency for the post. They may show it, but only within a totally different technical stack. Smaller companies tend to have less capacity and time to get people up to speed with new tech. So will likely fail these candidates even though they are capable of doing the job eventually.
The technical interviewers will tend to pair on the assessment - to improve its consistency. Swapping partners for interviews regularly also helps.

The assessment process is likely to use some online system such as Jobvite or Greenhouse where each interviewer assesses the candidate. Finally summarising it all with a recommendation for strong pass, pass or fail. Sometimes for a specific post and grade, otherwise the assessment can include a grade recommendation. The manager then rubber stamps that assuming appropriate funding is available. HR's job is to beat the candidate down to the lowest reasonable price, without going so low the candidate walks away.

A healthy growing company will tend to have a rolling recruitment process as they expect to be increasing head count in proportion to customers and revenue. On that basis they will likely be aiming to recruit anyone with a good pass, plus maybe most of the passes too.

Given that engineering jobs are highly specialised and require relevant experience I have not seen cases of way more interviewees than jobs. Currently, even with all the redundancies, there is still an under supply of engineers.
Also the approach for HR will be to set experience and skills pre-requisites for the roles that will keep the numbers down for those who make it to technical interview to around double the number of vacancies. Since it takes out a day of work for each interviewing engineer, to prep, interview and assess.

HIRING SUMMARY

You must pass each of the first 5 or 6 steps to get to the next one and get the job.

1. HR check written application is a plausible candidate
2. FAANG sized company - automated quiz / Leetcode style challenge - to reduce the numbers - because they get way more speculative applicants.
3. Recruiter chat to check candidate is genuine and available
4. CV skills / experience check vs other applicants to shortlist those worth interviewing
5. Technical task could be takeaway or whiteboard / questionairre interview
6. Technical interview, in person or Zoom with engineers
   
   
7. Manager interview / introduction to team mates. 
8. Recruiter chat. Negotiate exact salary. Agree start date.
9. Contract is signed, YOU ARE HIRED.

Ten Software Engineering Managers

Engineering management from the perspective of the managed.

I have worked in the software industry for many years. Working in both the public and private sectors as an individual contributor software developer, SRE and cloud engineer. 
Along the way I have been managed by 15 different managers, along with having work interactions with around another 100. So naming no names I thought maybe it is worth distilling my managed life into a set of software manager caricatures.

To illustrate what makes a good software manager (and a bad one).

I accept that since I have never chosen to become one, then I am criticising a skill I have shown no interest in acquiring myself. I have only partially dabbled in it, via senior IC roles, ie Staff Engineer technical advocacy. However it is a good chance to let off steam ... and possibly a software manager may read this and reflect on which characteristics they may want to work on. So guys its time for your 360!

Remote managers are better

Having worked for many years in offices with my manager sat at the desk behind me, literally looking over my shoulder. I should probably also admit that in recent years I have chosen to work full time remote, ideally with my manager in another time zone, or at least another country. Luckily I got on with those managers that were literally breathing down my neck. But it certainly didn't help me get the job done.

Full time remote is probably not so good an option for those who are just starting their career. However for established engineers it does tend to insulate you from the various forms of toxic management out there and lets you get on with the job. It also requires you to be more explicit about the engineering process, collaboration and documentation, and hence be more productive in a more maintainable manner.

Bullying, micro-management, under-mining, overly personal etc. Although these can all still happen in front of colleagues on Slack and Zoom. But it is easier to shut down a conversation and walk away virtually so that the manager has time to calm down and control their behaviour. 

Manager's skilled at managing remote international teams have to be skilled at targeted succinct and effective communication. Especially if they only have a few hours in the day when the team members time zones overlap. 

On average I have found remote managers to be better managers. Although that may just be that the UK is renowned for bad management - both anecdotally and in terms of the UK's productivity ranking. So managers that are from more productive countries than the UK are likely to be from a better management culture - hence better managers.

Peer level managers are better

In a traditional hierarchical organisation. The CTO is the most senior manager and so remunerates people more who are managers like themselves. Particularly in the public sector there is an unwritten rule that even the most junior manager must be paid more than the most senior engineer.

This approach naturally leads to some rancour amongst senior technical staff who want to stay technical. It also devalues technical skills. Since to increase their salary technical staff must eventually give up all their years of technical skills and somehow gain 10 years of skill in people management overnight. Of course this doesn't happen so instead you get very novice people managers with a lot of largely irrelevant technical skills, and perhaps a personality totally unsuited to enabling their team.

It is easy to filter out such an organisation. Just get on one of the company feedback sites, eg. Glassdoor, Fishbowl etc. Check what the top IC engineers salary range is and check that it is higher than the junior management range. Ideally you should expect the top IC grade, eg. senior technical architect to be paid around 50% more than an engineering manager. But take it as a red flag if there is no technical grade that is paid more than any management grade.

Because it means the organisation doesn't value your engineering skills, and you are likely to be managed by someone who doesn't value them either and may regard themselves as your superior. So why would you work there? Surely better to work for a manager who treats you as a peer in an organisation that values your skillset.

Most people quit their job because they have a bad manager

The surveys tell us 43% leave their job because of a bad manager. With the next most important reason being general toxic culture / under appreciation. Whilst pay and progression comes in third.

We all want to get the number 1 manager, but unfortunately we often end up with 10.

So most people leave their jobs because of getting one of the bottom ranked manager types.

HR's job is to minimise disruption to the company, they will tend to take the side of the more senior employee unless that employee's behaviour is clearly proven to be detrimental on the wider scale to the organisation.

A few junior employees reporting them for incompetence or abusive behaviour, does not usually qualify.

So if you do complain then it is unlikely to improve the situation, and may well make it worse if the manager is informed that you complained about them. Sadly I have not personally heard of any case where complaints about bad management resulted in resolving a problem, but definitely have heard of cases where it made it a lot worse for the remaining time before the complainant's notice period was done.

So unless a more senior member of your organisation is on your side, and decides to deal with the problem manager. It is best to leave your job if you have a bad manager. For a big company that may just mean changing departments. But leaving your job is safest in terms of removing yourself from a toxic environment, plus you can honestly report the manager as being the reason for leaving in your exit survey. It is a chance for you to help your ex-employer, you shouldn't expect them to act upon it immediately. But given a sufficiently high attrition rate for a manager's team, the higher level managers should hopefully have enough sense and self interest to deal with their failing colleague.

Ten Software Managers

1. The Enabler 

The ideal technical manager is the enabler. They have better personal skills than software skills.
They are most likely to either have never been an engineer, ie they entered the industry as a professional technical manager. Or they were an engineer some years ago but are more interested in the people than the code so changed direction fairly early into their career.
They will be well aware of the wider environment of the organisation and the stakeholders and drivers for the technical work. A great communicator. With knowledge of all the systems and processes and so how to unblock things and get stuff done procedurally. Plus be a talented scrum master.
 

Most likely to say:  How should we fix that process for you?

Catchphrase: Thanks for your contribution 😊

2. The Big Cheese

You may find at some point in your career you get a manager who is actually a much more senior person in the organisation than their surface role as an engineering team manager. Maybe they founded the whole company, or they are the head of large department with other non-software engineering work.

To be high in the organisation they are likely to be a better manager than the average manager you might expect, with great people skills.

(Obviously this rule may not hold once you get to CEO level, I hope nobody who ended up with Elon Musk as their direct line manager is reading this ... although I don't think he was interested enough in software to directly manage software engineers, given he gave up coding at 20 before getting any formal training in it.)

They are likely to be more of a leader than a manager, but likely to be particularly good at fostering the development of their engineers. They are also going to have all the contacts to unblock any issue that may arise, plus have their finger on the pulse of high level changes and strategy that may impact your job.

They may still have a surprising high level of technical understanding of the company's software, but as a senior manager they also understand that their job is all about enabling an even higher level of understanding, and technical decision making about it, in their engineers.

On the down side, they probably don't actually have that much time to devote to you personally, so don't be surprised if they fail to act on things you have suggested to them. If there is any issue they make damn sure somebody in the team takes responsibility for sorting it out. So be prepared to be volunteered by them.

Most likely to say: How are you aiming to progress in the company?

Catchphrase: Isn't it amazing what we are building at (fill in organisation name)? 🏆

3. The Bro 

They were happily doing their engineering job when the manager for their team left. They were not necessarily the most technical member of the team, but they were the one who got on with everyone and the one that everyone in the team was happy enough to have as a manager. So they took the job.

They want to be you best friend and genuinely aim to protect you as a member of their team from problems or issues that are coming down from higher up the management hierarchy.

They are reasonably technically aware and skilled, but don't try to make the technical decisions or deal with issues unless nobody in the team steps up to the plate, then they take the task on themselves.

They are just one of the gang, but your manager too.

Most likely to say: Let me buy you a beer ... umm, sorry, everyone in the team has been made redundant.

Catchphrase: Yeah, what the hell are management up to 🍻

4. The Middle Manager

They used to be a techie, many years ago, but they weren't really interested in tech and hence were probably pretty mediocre at their job. But thankfully they got on the management track, result! They love the politics and intrigue of management way more than technical details. They have good people skills but find it hard to hide the fact they have absolutely no emotional investment or interest in technology.

They literally couldn't care less if the head of the organisation declared that from this day forth all software in the company will be written in Cobol and all open source was banned from use. If that is what their boss says, then their job is to listen to their team moan and complain about it. Then tell them that is what they will be using. Since any technical objections the team gives are meaningless to them.

On the plus side they do not micro-manage and they appreciate their team members skills, and are good at bringing those skills out.

The middle manager likes people and wants to please them. But knows that their job is only to please their superiors. From their team they just need compliance, and being good at their job is all about bringing their team onside, with whatever the higher ups require. However outlandish.

Most likely to say: Sorry it wasn't my choice, but come on, we need to get on with it.

Catchphrase: I raised your issue with senior management, but no go. 😢

5. The Invisible Man

The invisible man works in a big organisation and knows the value of his super power. He used to do a bit of useful work, but that was years ago, when he still took enough interest in his job to get to the bottom rung of the management ladder. But over the years he realised he could get away with doing less and less actual work. He mastered quiet quitting years ago, before it became a thing. 

Since then he (and his boss) has worked out that if he gets a team of reasonably senior self starter engineers, then they don't actually like or want to be managed. So a difficult opinionated team for some managers, are actually perfect for the invisible man. Ideally if they are a distributed team, then he can "manage" them remotely, with the minimum level of work. Send the odd email maybe, do one or two Zoom calls a week and his work is done.

His team may not respect him, they may even play jokes on him. But he so doesn't give a crap about work that he won't even notice them doing it. As a manager the invisible man is the mid-point of the top 10.

He is neither a good or bad manager, he is like having no manager at all. He will never support, challenge or rebuke you. At least nobody every quits their job because they have the invisible man as their manager.

Most likely to say: Sorry just had to step out for a minute.

Catchphrase: Keep up the good work guys. 👻

6. The Over Employed

The over employed is a people pleaser. They like to say yes to everyone, including their managers and their team. They like to be seen dashing about doing things. So sure they will do that for you tomorrow ... but tomorrow never comes! Because the over employed is too busy. They may even have got themselves a second job on the sly, thinking they can juggle both at once. They are such an optimist, of course it will work out. 

They carry on saying yes to all those tasks that you need them to get done, to unblock your work. Just as they do to everybody else. So sure, they will sort out your performance review. Talk to the other manager that you need info from. But somehow they never seem able to deliver on time, if at all.

They will be there at 3am the night before a major deadline, chucking something together that is not quite finished and misses some vital component. But it is good enough, should do what is needed.

Poor people pleaser working their arse off to please people, so why is nobody that pleased with them?
But they stay cheerful,  not going to let those moaners drag them down.
Oh well if colleagues get too annoying they can always bail out. Get a job somewhere else and leave behind all those trivial little tasks that people kept bugging them with.

Most likely to say: Yes sure, I will do that.

Catchphrase: Hiya guys, why the long faces? 🤯

7. The Team Player

For the engineers in their team the Team Player is one of the best bosses they have had. They always have their back, supports them and persuades those above him to funnel more resources and authority to their team.

They are ambitious and aiming to rise higher up the management tree, but loyal to their guys. They know their team is really the only one in engineering that is run properly. It is also the one doing the work that matters. They makes sure they sweet talks those above them and dedicates a reasonable portion of their time to making sure they know that they and their team are the keystone keeping the organisation running.

They know that anybody looking to advance their career should be spending a good portion of each working day working on their own personal advancement. Don't be the fool who spends all day every day just doing the organisation's work.

Unfortunately their highly competitive nature and self belief, can lead to self delusion. They start to believe their own self promotional narrative. This leads them to be contemptuous of those annoying flakes in other teams who are not doing anything of real value. Though generally energetic and positive those outside their circle and below their grade, get the aggressive, bullying, dismissive and unpleasant side.

This behaviour also distorts the true importance and funding that the organisation should be devoting to their team's remit, to the detriment of other areas. So can cause problems for the company as a whole, as well as for morale outside of their team.

Most likely to say: My team has got that, we will save the day.

Catchphrase: Get out of my way, unlike you, I have a real job to do. 💪

8. The Bureaucrat

Once upon a time, long long ago. Some software companies decided they wanted to sell their wares to the ancient hierarchical institutions of government and learning. Those institutions believed in traditions and rules and processes and making things quantifiable. So something that did that for software management was perfect. So the companies came up with traditional names signifying regal wealth and power - Prince. Along with naming their software, Enterprise software, signifying new, wealth generating and boldly taking the initiative. That was in the 1980s.

It was the perfect sales pitch for these outdated institutions and they bought into it wholesale. Although it took them about 25 years to get around to it, old institutions are like that.

Their procurement processes and software and its life cycle and management were bound into reams and reams of bureaucratic processes. The IT managers in those institutions were groomed in the ways of Prince 2 and ERP and ITIL and all the rest of the snake oil the companies had invented. They devoted all their money and time to the training and meetings and processes around it. 

As far as the engineers in those institutions were concerned, a few of the processes were useful but that was far out weighed by the whole bureaucratic burden and costs they were wrapped in.

The manager spoke at length for years at far too many meetings of the process and the newly procured systems, but unfortunately the quality and features of the institutions systems seemed to have gotten worse. Whilst the cost of them became much much greater.
The institution employed more and more managers, although they just managed projects not people. But eventually there was so many, they needed managers for them too.
But they hired no more engineers.
Eventually the institution decided it didn't really need any in house software engineers at all, why were they writing software when they should be buying proper Enterprise software from the companies?

The engineer realised they had to leave the kingdom of the Prince 2 and its manager and go to live in a different place altogether, where their business was making software. Strangely in the software republic they had never even heard of Prince 2. They vaguely knew of PMP, but nobody in their right mind would use it to make or manage software.  

Most likely to say: I am sorry I cannot talk to you, until you have filed a change request.

Catchphrase: Our KPIs show that we are on course for all our CSFs 👑

9. The Technical Superior

The technical superior is at least a grade or two above you and always interacts with you as your superior.

They were an engineer and still secretly preferred being an engineer to their current job. They were never the best technical engineer in a team, so they compensated for that by imagining they were the best at seeing the big picture engineering wise and still are. So they decided to become a manager to make sure the right technical decisions are made.

 
They probably preferred their old job because they didn't have to spend so much time on politics and relating to people. They have been a full time manager for at least 5 years and their technical knowledge and judgement have dated badly. However they still see themselves as the person most qualified to make technical decisions. The more senior they become, the more out of date their technical knowledge becomes, yet the bigger and more expensive are the technical choices they make for their organisation. 

Most likely to say: Never ask a bunch of developers to decide on technology, ask 5 and you get 5 different answers.
Catchphrase: We really need a big monolithic Java XML SOAP web service to do that. 🙈

10. The Rockstar Techie

The very worst engineering manager is the rockstar techie.

Rockstar techies have great technical skills and may have technically saved the day a few times for senior management, so their lack of personal skills are overlooked. Plus they tend to behave much better towards people above them in the management hierarchy anyway.

But in the long term they are damaging to the quality of your codebase, the more senior they become the more damage they do. The common technical issues they can cause are blocking devolution of architectural decisions and diverse input into them. Possessiveness over code, or technical knowledge. Wanting to be the saviour for technical problems, outages etc. 

However the damage they do to the code is minor compared to the huge damage they do to the engineering team and culture if they are put in a management position.

They were often the most senior technical engineer in their part of the company. They have probably been there a while and to justify getting another pay rise they got lumped with doing management too. They regard management as an annoying burden tacked on to the side of their real engineering job. They aim to remain the lead engineer in the team and make all the final technical decisions. They cannot devolve technical authority and have no interest in picking up any management skills. Leading to exhibiting basic level fails in terms of interpersonal skills, have a technical superiority complex, be rude, moody, bullying, micro manage etc.

 
As time goes on they must devote more and more time to management and yet cannot accept no longer being the most technically adept guy in the room. A paradox that can only be solved by driving away any members of staff who challenge them technically and effectively dumbing down the technical skills of the whole team.

Most like to say: You Fffing broke it you moron, when coming across a feature change implemented in a way they didn't expect.

Catchphrase: You are wrong, this is how it must be done, idiot 👹

(Any resemblance to persons living or dead is purely coincidental)

Tech sector lay offs

INTRO: Having failed to post any technical articles for a few years I feel that my blog is at risk of dying from lack of attention. So to avoid that,  I have decided to mix up its content a bit and diversify from long Technical HOWTOs to more casual short posts whose tech content may vary (or not exist at all) ... so to kick off this one is a short rant about tech news!

Fake News about Tech Industry Collapse

A number of bigger tech companies are laying off staff at the moment. The press reports this as a related to them making terrible errors in misreading trends post pandemic and suddenly hiring way more staff than they normally would over the last year, 2022. Now reality has hit and the tech sector is awash with newly redundant workers as big tech desperately tightens its belt to survive. But is there any truth in either the premise of this argument or its conclusions?

A random recent example that repeats these ubiquitous assumptions ... https://fortune.com/2023/01/23/big-tech-layoffs-15-20-percent-next-six-months-top-analyst-says/ ... Citing the not for profit basket case, Twitter, being turned into a uniquely loss making zombie company by Elon Musk as though Google, Amazon and Microsoft had something in common with it as a business!

If you look at the graph of the employee count of these companies year on year. Then only Microsoft hired more than usual last year, 2022, and Amazon to cope with its pandemic boom did so in 2020. Google last had an uncharacteristic hiring spree in 2012. Similarly Facebook and other big tech companies growth curve exactly followed that of the last decade in recent years. There was no extra hiring.

So why the layoffs. Nothing to do with over hiring. Simple - look at the share price curves instead.
The market is valuing most big tech lower and recession looms - they need to chop staff to chop costs and make their finances look better to reduce that drop for their shareholders - the biggest of whom are the CEOs of those companies.

The big companies are still making billions in profit (they are not loss makers) so over the long term it would cost them less to retain talent, and they can afford to. However the CEO's personal short term loss in wealth is something that they can't stomach, and it is a good excuse for a clear out.

Obviously there are large loss makers, the most prominent of which is Twitter, but they are special cases with failing business models - Twitter only ever made a profit in the run up to Trump's election as it became a huge engine churning up ideological conflict with political and conspiracy fictions. Without a politically polarised USA to drive an explosion of lies and social media wars, it was always a loss maker. It has nothing to do with big tech trends. 

Apple is a real exception to the trend, currently, but only because its share price hasn't dropped significantly yet. Hence it hasn't done its layoffs yet.

So the real reason big profitable tech is laying people off is a temporary fix to save a few billions from the bear market's current swipe at the personal wealth of their CEOs. Even though being a slave to short term market driven fire / rehire cycles will cost the company more in the long term. It is purely a personal choice to save personal wealth, there was no over hiring, there is no need for redundancy, there is no down turn in the growth demand for the tech sector, there is just less cheap loans around to fund it.

 
The jump in the cost of borrowing is due to what looks to be a short term hike in interest rates, nothing to do with the tech sector itself. Largely due to Putin starting a war of revenge against his long dead ghosts from 35 years ago, when the Soviet Union fell as he presided over the KGB. A war against ghosts can never be won ... but unfortunately its far more terrible human cost will carry on for years.

This becomes clear on the other side of the market coin, late stage startups are often making no profit at all - because all profits along with loans are ploughed into growth. Because they must never be seen to be shrinking - to keep growing their valuation for IPO.

So a lot of software companies are hiring even though strictly they don't need to, whilst the big boys are firing when they don't need to.

Meanwhile the number of software jobs as a whole keeps growing at 10% year on year. The demand continues to outstrip supply and wage inflation follows. So if you are a talented (if overpaid) software engineer ... I wouldn't worry too much about the layoffs. It is just a chance to take a redundancy bonus and a 20% pay rise to try something new. Unfortunately for those that were used as a disposable foreign human resource by big tech, via job dependent visas, it is a different story. They may well not have time to stop their CEO's thoughtless greed needlessly disrupting their lives.

Many predictions are that interest rates will drop and the bear market end in a year's time. At which point the mass layoffs will be reversed. But the big tech companies will have lost a lot of money, and a great deal more trust, by following the market and each other so closely. The lesson employees will have learnt is that loyalty to such companies will never be rewarded or returned.

Using Java Spring & MyBatis for dynamic schema integration

Using Spring MyBatis for dynamic schema integration may perhaps be subtitled, "Eating soup with a fork" ... since that is how it has felt at times. However that may partially be due to my lack of familiarity with these tools. So this blog posting is about why I got that feeling, how I bent my fork to the purpose ... and if I missed some tricks in doing that. Please let me know by commenting below :-)

Object-relational mapping - a personal history

I have been developing relational database applications for many years so I have been through the various stages of database persistence approaches. Starting with fairly raw SQL development or at most just fixed DAO wrapping. Through SQL templating languages and mapping approaches to largely adopting the full object-relational mapping, ORM, approach for the past few years. I even tended to use the same pseudo ORM top level for NoSQL sources, although of course a lot of ORM features then become inapplicable, such as use of an ORM model's relational navigation (eg. person.department.head.surname)

Having said that the object-relational impedance mismatch can still bite. However good an ORM is there is some need for understanding of what its API simplifications and abstractions are doing. To avoid excessive  / badly performant SQL. But given that caveat, the amount of time saved in development and maintenance by a well implemented full ORM inevitably saves money on all but the simplest persistence requirements. Similarly the automation involved in schema migration, constraint, indexing and other data modelling needs is invaluable. As long as one knows what it is doing under the hood and hence when that 10-20% of database development / customised data design is required. For example custom functions and triggers, data writable views that present as a table for the ORM, etc.

MyBatis - is it an ORM I see before me?

Recently I have started working with a team who use an older Java technology set of Spring and MyBatis. My current project uses the Java MyBatis SQL framework to integrate various relational database sources.
MyBatis badges itself as a persistence framework. So it is not really claiming to be an ORM. It started as an SQL templating engine and that is still where its core skills lie. But it now has many other features bolted on. Including meta programming to generate Java relational mappers from database sources.

So it does have many ORMish features. However using MyBatis has been rather a painful journey for me, since I have found that it fails to implement many ORM design pattern principles that my ORM habituation had come to expect. Also using MyBatis with Spring MVC means we havent got proper MVC since the model is not a model as it would be with Hibernate ... its a MyBatis Mapper.
This is not just a minor niggle, on first use a core ORM feature is gone. A Mapper is not an object model of a table's row of data, with transaction and session management under the hood. Instead it is just a convenient object to hook up standard data queries and updates to with the management of saves and synchronisation of actions on Mappers and results returned by them, entirely manual. So in that sense you have less than what some more basic DAO wrappers give you, ie. when you update a record then fetch the values from that record they will return the data written into and queried back from the database, hence an inserted saved model would automatically hold the new primary key(s).
With Mappers the data readable from the Mapper will just be the data set to be updated not the actual data held in the database as a result of that update. To find out the result of your query you need to ensure you get a fresh query result, by requerying or flushing session.

There is a work around for this common requirement, you can annotate an insert with a custom select query to tell MyBatis to re-query just for the key manually before or after doing the update. However I found this didn't work well for Oracle and was a problem anyway with what I required. Largely because I was using Generator based classes, so adding custom annotations to these would not have worked for me.

@SelectKey(statement="SELECT MY_SEQUENCE.NEXTVAL() FROM DUAL", keyProperty="nameId", before=true, resultType=int.class) 
int insertMyTable(Name name); 

Similarly there is no lazy update concept, updates are only and always done when you specifically run them. In effect its like using an ORM but disabling all the relational modelling, data synchronisation machinery and transactional management. Mappers are instead just convenience wrappers for running SQL - MyBatis Generator may do the ORM process of creating Java classes that (via XML) map to database tables or views. But mapper instances are not ORM synchronised table row objects.

The normal object centric approach is also not available. Whilst you can generate code from your database, you cannot generate your database from code. So full persistence data cycle management is not available out of the box. By that I mean the use of test harness and deployment tools that destroy and recreate the full database, populate it from fixtures, run tests and drop it again. A migration tool that introspects databases and can migrate back and fortbr /br /br /h between schema versions to sync them with code versions or vice versa, ie code -> data, schema management, in addition to doing the MyBatis Generator data -> code direction. So the sort of thing that a full stack web framework like Django has with its ORM.

This was a blow for my particular project since full data cycle management is exactly what I required for developing a central data aggregation and integration database. I also required proper fixture management for testing and the same serialized data input / output components of this being used for the main work of data aggregation from data sources in serialized format (XML, CSV etc.)
Finally the system was dealing with pulling in and delivering back out complex, evolving schemas from source to consumer databases. The one thing that would be essential is that all the schema handling and data typing be dynamic so that the code can handle this change rate without constantly breaking and having to be rewritten. So hard coded schema details and static typing must be avoided for any chance of maintainability. But it was to be written in a statically typed language. Again the tool was not suited to the job. So how did I bend this seemingly unsuitable platform to the task in hand.

Bending the fork

The first step was to tackle the dynamic schema vs. static typing issue. Java reflection is your friend here. Then reinvent some of the missing ORM wheels. Fixture loading, schema generation.
Along with working around the lack of object data synchronisation and automatic session management. Finally we wanted to make our data population speedy by only updating data that needed updates. So a hashing mechanism that could work with the serialized data was required.
Note that our in house Java standards decree that all configuration should be done via annotations rather than XML where available ... so that is pretty much everything in Spring ... aside from the core MyBatis Mappers XML files. The aggregation database was Oracle and various database's including MS SQL Server were the data sources.

Dynamic schema handling

Use MyBatis Generator, adding a config file for each database source and the target aggregation database.
Add the name of each table or view required for them to these config files ... eg.

<table tableName="PERSON" modelType="flat"></table>

ideally that would be all the schema customisation required as long as data source and target naming of tables and columns matched. However in a handful of cases mapping data (eg. tableA.colB = tableC.colD) was needed in the properties files.

I ran this as part of the Maven build at first, but it proved more manageable to be able to run code generation for the sources and target databases at different times, so  I switched to wrapping this up in a command class for triggering independently via a batch job scheduler. In the same way as the data population commands were to be run. Schema -> code generation tended to be much faster than either data population or code -> schema generation, so freqprepreuent execution was not an issue.

The core class I wrote for enabling dynamic schema handling was called TableMeta and each table has an instance of this class listing all the metadata about its columns and primary keys etc.

This class was injected into a ModelFactoryService whose job is to return results or add, edit and delete models from any Mapper.
The Factory Service has a reflection based method for invoking the MyBatis generated Mapper methods, e.g. "SelectByExample"  ...

  protected Object invokeMapperMethod(String methodName, Object parameter) {
    Class klass = exampleClass; 
    if (parameter != null) {
      klass = parameter.getClass();
    } else {
      parameter = newExample();
    }
    try {
      Method method = mapperClass.getDeclaredMethod(methodName, klass);
      try {

        // Run the method  
        return method.invoke(mapper, parameter);
      } catch (IllegalAccessException | IllegalArgumentException exRun) {
        Logger.getLogger("ModelFactoryService").log(
         Level.SEVERE, modelClass.getSimpleName() + "." + methodName, exRun);
      }
    } catch (NoSuchMethodException | SecurityException exCall) {
      Logger.getLogger("ModelFactoryService").log(
         Level.SEVERE, "Mapper class has no " + methodName + " method", exCall);
    }
    return 0;
 }

... similarly there were reflection based set and get column methods surfaced by whole Mapper data modification methods such as doUpdate(Object model)
The TableMeta caters for initial setup of the classes generated for that particular table by MyBatis Generator. Straight use of Class.forName(className) works with manipulation of the table name string copying that of the Generator's standard naming convention. The table's Mapper, Model and Example classes are then added as the template classes for the Factory. From that a Setters and Getters hash of the table's Mapper methods for each column can be built automatically.

Fixture loading

A sax parser based loader class reads in data fixtures in XML format. Each fixture row is parsed to a hash of tag name to value which can be passed via a SaverService to the ModelFactory.
An ObjectConvertor static class caters for de-serialising fixture data to the correct Java types looked up via the column name from TableMeta, and the Setter hash can then be used to update the matching named columns in the model.

Fixture dumping

For fixture dumping I cheated a little. Since I was only ever dumping from the target Oracle database then rather than reinventing a full serialisation of models (e.g. ORM with built in serialisation from any database to xml, json, yaml, csv etc.) I just provided a tool to do the minimum required for my testing and development needs - to just be able to serialise any table or view to XML from Oracle.
So in this case, to avoid the maintenance madness of a separate statically typed fixed schema solution for each table - I did the XML part in Oracle.
That way I could use a GenericMapper which generated two string fields based on a dynamic SQL query built from the TableMeta. The first being the row XML and the other the concatentated list of the serialised columns to be contained within it. MyBatis @SelectProvider annotation allows the gluing on of a method taken from another class to generate SQL ...

 @SelectProvider(type = TableDump.class, method = "getXMLProvider")

 List<GenericModel> getXML(@Param("meta") TableMeta meta, @Param("rows") final int rows);

The method glued on is this one which uses Oracle's native XML methods and the TableMeta's list of columns to generate a query that directly returns the XML..

 /**
   * Dynamic SQL generator method generates XML output for fixture as row and cols 
   */
  public String getXMLProvider(Map params) {
    final TableMeta meta = (TableMeta) params.get("meta");
    final int rows = (int) params.get("rows");

    String sql = new SQL() {
      {
        String rowXML = " '' || XMLElement(\"row\", XMLAttributes(";
        for (String col : meta.getPKeys()) {
          rowXML += col + " AS \"" + col.toLowerCase() + "\", ";
        }
        rowXML = rowXML.substring(0, rowXML.length() - 2) + ")) \"ROWKEYS\"";
        SELECT(rowXML);
        String queryXML = "''";
        col_loop:
        for (String col : meta.getCols()) {
          queryXML += " || XMLElement(\"" + col.toLowerCase() + "\", " + col + ")";
        }
        SELECT(queryXML + " \"ROWCOLS\"");
        FROM(meta.getTableName());
        for (String col : meta.getPKeys()) {
          ORDER_BY(col);
        }
        if (rows > 0) {
          WHERE("rownum <= " + rows);
        }
      }
    }.toString();
    return sql;
  }

This above snippet shows an example of MyBatis SQL templating - the core of MyBatis.
The result is a single Generic Mapper that by calling with the appropriate tables TableMeta can return an XML dump of any table in the target database. For test fixture generation this can be passed a result length ... since usually ten or twenty will be sufficient for integration testing.

Save methods

I won't go into the save methods in detail - suffice to say the data was hashed on the way into Oracle and then a query of primary keys to last modified hashed data returned of all rows to allow incremental update by hashing each of the XML data sources rows and checking it first to determine if an update or insert was required. Whilst this could not take advantage of databases or Java's hashing - since it had to be serialised data compatible - it did have a significant impact. Since the rate of incremental updates meant that we are getting a maximum of 1% data churn per table, so even with all the hashing and comparision overhead - an incremental update is still around 30 times faster.

For versioned data a core data table and a versioning table was required and this then needed both to get access to the new primary keys. As mentioned this is not straight forward in MyBatis - so the easiest solution was to have a bespoke version Mapper that just wrapped a single versioning sequence and could be used to update the related data and version tables by calling custom VersionMappers nextVal or currVal methods.

Code Schema Cycle

As mentioned MyBatis doesnt cover the code to schema half of the persistence cycle, so the option was to employ a separate full schema life cycle framework such as Liquibase. Or roll my own. In this case my requirements were not for extensive migration features. Since as an aggregation database the schema could be snapshotted dropped and rebuilt. So to avoid complexity and further dependencies I just added a tool to build Oracle schema from standard dumps of the schema from an Oracle client tool. So you just dump each object to a separate DDL SQL file, then it checks all the files works out which object type they relate to and loads them in a sequence which should prevent referential integrity clashes, ie.

DATABASE LINK, SEQUENCE, TABLE, MATERIALISED VIEW, FUNCTION, VIEW,  INDEX, PROCEDURE, TRIGGER, CONSTRAINT, FOREIGN KEY

This is wrapped up as a DatabaseReset command that either drops all data or the full database and rebuilds the schema. So a run of  reset followed by the GenerateMappers command gives a freshly built persistence layer and Model code to talk to it. It would be nice to have the schema built from the Mappers code and a more standard coherent code schema cycle, but given Mappers are in use, that would not be available even if Liquibase were in the mix.

Spring issues

So as a newbie to MyBatis I guess I had some problems with it. But as long as this slightly irascible post is. Guess what, I also had issues with Spring too ... so I will make it even longer by getting those off my chest too :-)

It seems to require huge amounts of configuration to do what you want. Again it can be made to do most things, but ... and maybe to some extent ... because of that ... it takes an awful lot of configuration to do some of the things a more opinionated full stack web framework would do out of the box. I guess I need to come across the sort of edge cases which are really difficult in a full stack framework, to appreciate Spring, since currently it feels like it needs a great deal of maintenance heavy tinkering to do some of the basics.

So the first surprise was that all my Spring @Service classes default to singletons. So in order to have for example a ModelFactory for the two different tables a versioned update required I needed to build a BeanFactory and make these beans and annotate them as prototypes - ie normal classes not singletons. I guess this is because all of this related to command classes not Spring MVC web classes ... which would have had web session scope.

  @Bean
  @Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
  public ModelFactoryService modelService() {
    return new ModelFactoryService(prop, hashService());
  }

So all my indvidually injected service classes tended to have to be uninjected and instead the bean factory injected in their place.

Along with that the database session to mapper class connection seemed rather frail.
The only way to ensure this worked was never to use a generic session handling method but always use separate prototype beans for separately named session classes annotated to find the correct mappers for that particular database ...

@Configuration
@MapperScan(basePackages = "uk.ac.bris.adt.erp.dataint.sources.mm", sqlSessionFactoryRef = "MMSessionFactory")
public class MMDBConfig extends DBConfigABC { ... }

... not only that the directory hierarchy matters due to the niceties of MyBatis-Spring , since the MapperScan will find anything at or below a directory ... so you cannot put mappers for one connection below the point you need to scan for another ... or they will be sucked up as Mappers for the wrong session.

Test configuration

Also the test configuration for integration tests required a lot of manual annotations to pick up appropriate configuration environment in order to inject things into the context in a working manner to match the running code configuration. Again I had been spoilt by expecting a framework specific automatically working test harness with default test customisations of the runtime code environment thrown in. Instead each integration test needed to use a wrapper that called a RunnerBeanConfig

@PropertySource("classpath:test.properties")
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(loader = AnnotationConfigContextLoader.class, classes = ITRunnerBeanConfig.class)
public class ModelFactoryServiceIT extends ServiceABC { ... }

The RunnerBeanConfig then needs to have all the database and resources configs annotated as Imports or it cannot find the different database sessions. Only after all that is it possible to inject the runtime command classes for testing.

@ComponentScan("uk.ac.bris.adt.erp.dataint")
@PropertySource("classpath:test.properties")
@Configuration
@Import({ DataIntDBConfig.class, MMDBConfig.class ... ResourceConfig.class })
public class ITRunnerBeanConfig implements EnvironmentAware {

  private Environment env;

  @Override
  public void setEnvironment(Environment environment) {
    this.env = environment;
  }
  ...
}

Conclusion

OK so I got it working in the end. However it all felt rather like I was doing something rather bespoke and complicated. This added a great deal of development time (or more accurately configuration wrangling time) to deliver components of persistence layer management and testing - that I had previously expected would all be available already in a mature high level framework. So in summary I probably have to conclude that if you needed to do this task in a more maintainable and standardised manner with a minimum of custom code. Don't use Spring and MyBatis. However if you already use one of those tools ... and you need to add this sort of functionality with them. Then it is certainly doable, and perhaps if you are a newbie to them, like me, this post may be of some use in speeding the job up for you :-}