Lack of Success in the GCP Data Engineer Job Hunting Arena (but Success — AWS Data Engineer)

UPDATEs
Oct 2024: I did gain offers/jobs for AWS this year. May 2024: Booz Allen (mid-level data scientist) & Jun 2024: Deloitte (mid/sr level Data Engineer). See at the end.
If you desire to read the full paper, make sure you have coffee/tea & time….

There is also a dire warning about specific compliance requirements you need to verify BEFORE joining companies like Deloitte, McKinsey or other audit consultant firms.

Nov 2024: after resigning from Deloitte at the end of October — instead of looking for another Data Engineer job, I decided to return to completing my NVIDIA Gen AI certification as well as my AWS ML certification.
— — Then look for a new role in AI / ML.

========= go to end - update continuation to this Jul 2024 paper =========

Wow! and Holy moley… (if you’re in that camp)

Talk about disappointment, the job market is DEFINITELY tough right now for GCP (google cloud platform) & AWS Data Engineers overall. Tough AND Competitive….

So far, this Data Engineer (DE) journey has cost me a VERY significant amount of money for the past year (no salary and living on savings in our home). All in the belief in my self-confidence (which I very much still have) of getting into Data Engineering, see below.

The oddest part for me though, ~85% of the emails/calls I receive, they are for Senior or Lead DE and/or Data Scientist roles. It is very tough (and disappointing) for me when I have to tell the recruiters — I am trying to break in at the mid-level DE role because I know I do not yet have the Senior level experience yet.

• But give me 6–8 months and it will be a different story…

Do not get me wrong, if the companies find someone more qualified and at a lower salary, good for them. Just not good for me — that is the way it goes. All one can do is — keep on pushing, someone with vision in some company will see the drive and sacrifice (more below) you have endured / incurred / undertaken….

Yet, after gaining two of the ‘most’ desired DE certifications, I had believed I would be a shoo-in for that role, even with my ‘not’ yet having 2, 3–10 years ‘hands-on DE work experience’ most of the want ads indicate. When many of the folks working in DE do ‘not’ yet hold a single DE cert.

Certs obtained:

• AWS Data Engineering (Jun 2024)
• GCP (Google Cloud Platform) Pro Data Engineer (Sep 2023)

Or, it could be the lack of vision in the various company Personnel Departments (many people like the HR cattle term — human resources — I like to refer to people as people and not just resources). Or, even from the viewpoint of the lower-level team members who do the team interviews.

My wish is in actually talking with more senior leaders, those who have a much better strategic and / or tactical aspect. Whenever I’ve done that, I by far have more success.

Okay, the reason why I say this, is that it could be that some of these companies are just ‘NOT’ seeing my sacrifice and drive — which is as clear as day.

I took off a year to pursue the two DE certifications, when I had originally only meant to take off five (5) months, due to my high optimistic goal of success.

This time off from work stemmed from commuting for 2–3 hours a day in this region’s traffic — unavoidable. I had already done that kind of commuting for 15 years in multiple jobs in this area. And taking the metro-rail system as an alternative — nah…. It is equivalent or worse. At the end of the day, I was too burnt out to study every night and many weekends.

The best way for me to get the certifications — stop working to get it done faster. If I had not stopped working, I would still be working on earning the second certification. So, I stopped working and commuting.

I had believed after gaining the GCP certification and then spending three (3) months on learning SQL / Python, it would be a cake walk to get into the GCP world.

One seriously huge surprise I had not accounted for, was when I talked to a couple of Python experts, they told me that if I studied and worked for 6–12 months, I’d at best be at the Proficient level. Some folks would get lucky and become experts, depending on the work they did.

And then guess what, Covid slammed me down for over 3 weeks, bad timing. Lady Luck is having fun with me, lemme tell ya…..

Then, guess what else, the job market hit — big tech layoffs really affected my transitioning over to the GCP world. Note, I was trying to get a workplace close to home due to the pain of commuting here. And I wanted to be “IN” the work place and not fully remote — I prefer hybrid at best. But there just are not enough GCP Data Engineer jobs near me.

Because of the tech market slump, which we have no control over nor foresight in seeing ups and downs — I had to get back to work, so I jumped back to the AWS world where I had the most experience, to obtain the AWS DE cert.

My next surprise was when I took the AWS DE test the first time — I was shocked I did not pass. It was supposed to be at the associate level. So, I hunkered down again and passed it. The AWS cert is a very, VERY tough little nugget, especially if you do not have the 2–3 years of ‘hands-on DE work experience’ that AWS recommends — which I do not have.

But I passed. And have since started working on my AWS ML Specialist certification while looking for an AWS DE role.

At this point, it is very unclear to me as to what companies are looking at.

Or, it could be the lack of vision for many in Personnel or lower-level teams — my background in total is not being considered.

Nine years of working in the AWS Cloud Architect / Engineer space and holding these additional certifications:

• AWS Cloud Architect Pro (renewed Sep 2022, renewed, not a first-time cert)
• AWS Data Analytics Specialist (Dec 2021)
• ** While gaining the Data Analytics cert in 2021, I also stood up AWS Redshift for the Data Scientists on the team — had to teach myself some SQL / Python and pgAdmin to help the Data Scientist team (definitely thought this would aid in my Data Engineer job search)
• ** currently working on AWS ML (machine learning) cert

Also in the past year, picked up more knowledge in:

• pgAdmin
• VScode
• Jupyter Notebook
• Pandas
• Numpy
• Thonny
• and a little bit of LaTeX…

Double graduate degrees:

• MBA in Technology, focus on Strategy & Innovation
• MSc Cybersecurity

Multiple years of working in the:

• Intelligence space, where strategy/tactics are paramount.
• IT space — many years of IT pre-cloud: network / data communications, routers, firewalls, NOC & SOC — and being in leadership roles in many of them
• Cybersecurity space and holding these certifications:
• — — (ISC)2 CISSP (renewed multiple times, not a first-time cert)
• — — (ISC)2 ISSEP (a concentration, also renewed multiple times, not a first-time cert)

Communication and diplomacy skills — while not being the best writer or the best orator, I have engaged with many very senior leaders over the years:

• Ambassadors: In the past, I did multiple overseas visits to do one-on-one cybersecurity talks with several Ambassadors and their Deputy Chief of Mission
• Regional INS Border Patrol Commanders, the number one person for each sector, for cybersecurity
• Engaging in doing briefings, intelligence, with multiple executive leaders (civilian and military), while also letting various team members on my staff take turns to do briefings (so they can get experience) — one briefing included the head and deputy of an agency with only 12 of us in that, ummm, quiet meeting
• Engaged with all division level leaders at an agency to clean up their AWS Cloud assets and to stop wasting money every month (or course, it was not quite worded that way…) — saved that agency a LARGE amount of revenue
• etc., etc., etc…..

And let’s talk about cognitive diversity skills — my brain being one of damn good plasticity, the brains’ capability to learn more and do more… Always looking for better, creative, productive, successful outcomes.

The more diverse subjects/topics you learn, the better you become in learning and dealing with more challenging subjects.

Language skills, learned to read, speak AND write:

• Arabic (USC — adult learning, night classes)
• Russian (self-taught)
• Japanese (BC — college)
• Chinese (UW — university)

Music skills, learned to read Treble and Bass scale music to play:

• Trumpet (high school marching band/concert band)
• Piano (at UW)

Advanced Math in college

Physics (the version with calculus — many folks take the ‘non-calculus’ version of Physics)

Chemistry

Now, let me get the ‘possibly’ bad bullet out of the way:

1) Race — MOST of the time, this is difficult to determine

Then, there is the:

2) requirement of having 2, 3–10 years of ‘hands-on DE work experience’

• As a solid viable substitute, does it no longer hold water for achievements such as:
• — — holding double DE certifications

3) Salary

• my pay has been high for years but I am still willing to take a pay cut to get my foot in the Data Engineer door and have made that clear as well, just not too drastic of a pay cut — not with all the pluses in my basket.

So, in our current time period in the workplace, “does drive and sacrifice no longer hold any water?”

Especially for anyone who has stated to recruiters and the various companies that one is willing to come in at mid-level (in my case, it could not be junior level due to my years of AWS cloud experience and the degrees and the certifications) and do the grunt work for the first 6–8 months…?

What are recruiters and companies looking for? One can only do so much studying, along with some projects (I’ve read a couple of articles where companies do not care about the projects you did) prior to being hired. Especially when they need to return to work.

If a person’s drive and sacrifice is not a completely solid indicator that they want to do that work, what else could be a better indicator to bring that person in…?

This is a very interesting tech world we live in now. For the others of you having similarly tough progress, good luck in your journey.

I will guarantee in making myself valuable to any/every company who would want me. That is something you all out there had best work on, trying to figure out things on your own as much as possible — research. Besides, doing that research pays off in you covering additional grounds at the same time for tangential areas….

=========== update continuation to this Jul 2024 paper ==========

UPDATE (Oct 2024):
In May 2024, Booz (after 2 months of back and forth) did offer me a role as mid-level Data Scientist. But then, kicker here — they revoked my offer letter. Stating that I was a professional risk. This is because I had walked away from Booz in 2022 for an AWS Cloud Architect role after another company offered me $20K more for my education and experience for an AWS Cloud Architect/Engineer Tier III role.
I told Booz that I was not going to walk away from the Data Scientist role but, they pulled the offer anyway.

Then, late June 2024, as I was obtaining my second Data Engineer cert, AWS Data Engineer — Deloitte approached me for a Data Engineer role. And it was 10 minutes away from home (which was a FANTASTIC location). Then, it took 3 months for the onboarding and processing routine (later, some folks told me it took them 5 or 6 months for theirs). The long time was due to DHS’s clearance process.
This is even though I was cleared by DHS, years ago, before I switched over to intelligence (CYBERCOMMAND at NSA and then DIA (when DIA pulled all of us Intelligence assets under the DIA umbrella)).
At any rate, I came onboard at Deloitte and discovered that I was in a Dual role — Data Engineer and also mid-level manager for Cloud efforts that Deloitte has that (as I was informed) “I might be assisting on those projects”.

Now, the dire warning kicker — ONLY read if you are interested in possibly working for Deloitte.
Go over to my other paper on BDIPs and Restricted Entities at: my site or here on medium…
It is because of that Deloitte BDIP compliance requirement I refused to sign that I resigned from them, after only a month — at the end of October.

.

tags: acloudguru, aws certified data engineer, cloudacademy, data engineer, data engineering essentials, gcp pro data engineer, google cloud platform data engineer, aws ml specialist, aws machine learning specialist, drive, sacrifice, drive and sacrifice, cognitive diversity, communication skills, diplomacy skills, aws cloud, Jupyter Notebook, Thonny, VScode, VS Code, pgAdmin, Pandas, Numpy, ML, Machine Learning