HOT DAWG!!!! My SECOND Data Engineering certification.
To those folks who doubted me attaining both Data Engineer certifications:
SUCCESS…!!! — — Already started my AWS ML certification path… Decided not to wait.
Successfully gained the:
- AWS Data Engineering Associate cert
after studying my little hiney off for months. This comes after studying for months to successfully gain the GCP (Google Cloud Platform) Pro Data Engineer (Sept 2023) cert. This was to explicitly show that my mindset was/is in Big Data, for my path to Data Engineering and then to AI/ML Engineering (and maybe gain a PhD in this area later).
PAY ATTENTION: For those looking to break into AWS Data Engineering and obtain that certification - you 'CAN' do it. But… You definitely ‘should’ have at least 1–2 years of AWS work experience. And of course — hunker down and by that, I mean sacrifice your time for now and REALLY “HUNKER DOWN” and study your rear end off… IT WILL PAY OFF. And if you are like me, in being a visual, auditory and kinetic learner (I like to learn via multiple intakes) — you will be successful.
(heads up, several pages if interested….)
This amount of time I spent on so much studying (though I’ve been an AWS Cloud Architect/Engineer for a decade) was due to not having any kind of previous Data Engineer hands-on work experience.
So, because of that the path for me, to enter Data Engineering, was to:
- gain one (GCP) or both (AWS & GCP) Data Engineering certifications to make up for that “lack” of ‘hands-on work experience’….
- — — and…. I am highly, HIGHLY curious as to how many other folks out there have attained both of those two data engineer certs…???
- — — — — And what that successful outcome was like…??
A couple of years back, I had already achieved the AWS Data Analytics Specialty certification (due to commuting — “that one” took 8 months of studying: nights, VERY early mornings and weekends) — it did not help. DIA and the Intelligence Community had decided to go their own way and continue using their own silos (of whatever, can’t say) instead of collaborating together — and cancelled the contract we were on. It was the same month I gained the AWS Data Analytics cert….
However, it was a wee bit of a tough road — from Jun 2023 to arrive at this point now, Jun 2024.
NOTE: To do the equivalent studying of what I have undergone while one is working full-time, it would have been a plethora of pain. The studying period would last for many months longer, in attempting to obtain the:
- GCP Pro Data Engineer,
- AWS Data Engineer certification and a
- good solid foundational knowledge of SQL, Python (VSCode, Jupyter Notebook, pgAdmin) along with some Linear Algebra (refresher for me) and even
- a little foray into learning LaTeX…
In June 2023 I decided to make a HARD break to make time to study and obtain the GCP Pro Data Engineering certification and more SQL / Python knowledge — Only way to make more time was to stop working and stop commuting for hours every day. Thought this would be a successful and optimistically short break from working to do this.
At that point, I was commuting to Chantilly, VA from D.C. and in today’s world as opposed to 2005 - THAT-IS-A-VERY-TOUGH-SLOG every day. Especially when you are unaware of unexpected situations popping up in this region:
— — Freeway construction (pick a freeway here, any one or two of them are undergoing upgrades and whatnot at any time) — causing miles long backups and sometimes, they pop up unexpectedly — even at 5.30 am
- And if lucky, they end by 6–7 am for the other commuters. Sometimes, the work lasts throughout the day so if you are ‘very fortunate’, you see it AND go through it on the way home
- Taking side streets are not really an option for every one because if you did, it could add another 30–45 minutes to the drive
— — Unforeseen accidents — from simple to major and if are also lucky here too, it could last 45 minutes to an hour…
— — Rain or snow (and plowing has not cleared it yet) — these make for even more fun days of driving and HOPEFULLY you are not caught in flooded roads which happens once in a while (ask folks who use Clara Barton out in this area)
The point for that info above is — commuting completely burns out many of us, including me — making studying at night time even more limited and difficult to stay awake. Weekends are not enough because you have to complete other tasks that were put off during the week. And to spend “some” amount of time with your mate (cannot leave her out of the picture, every day).
After stopping work and studying for months, EVERY blasted day, I obtained the GCP Pro Data Engineer cert Sept 2023 and was HIGHLY optimistic in looking forward to gaining a Google Cloud Platform Data Engineer role.
Then, I took the time for the next several months to study nothing but SQL & Python to get better at it. After contacting a couple of Python / SQL experts, well — they kinda burst my bubble. They stated that even taking a year to study, one is not likely to become an expert but only reach a Proficient level…
At this point, it was November and guess what…. Covid knocked me out of the picture for several weeks.
Then another whammy hit at the same time — Fall tech layoffs…
There were not enough GCP Data Engineer roles to begin with for me. I wanted something close to home to avoid spending 2–3 hours on the road every day.
Plus, I did not wish to do full remote work because I wanted to be on site for the first 4–6 months to work with others and pick their brains to steal, ahhh, acquire more knowledge to help me grow. It is all about collaboration.
ADDITIONALLY, while I had a decade of AWS Cloud experience — I did not have 1, 2–5 years of GCP Data Engineer “HANDS-ON WORK” experience.
In Dec 2023, I stopped refreshing my Linear Algebra to prepare to return to work.
— In the end, my optimistic transition plan from AWS Cloud Architect (Tier III) to GCP Data Engineer failed — had to switch back to AWS hemisphere. Made that decision in early-mid January 2024.
Commuting to far away areas, such as: Ft. Meade (NSA) / Chantilly (NRO) / Springfield (NRO) are still not an option for me — I’ve been doing those kinds of commutes for a couple of decades now. And each year, they are becoming worse. Metro rail is not an option either — sometimes it might take as long or longer. Plus, the lines may not be anywhere close to the work locations…
So…. as of mid-Jan, started the, what I believed would be a relatively easy slog for studying in the AWS Data Engineer world.
After my past successes in previous multiple AWS certifications, such as AWS Cloud Architect Professional, and how easy it was for me to obtain the GCP Pro Data Engineer cert — I believed (truly) I should have no problem gaining the AWS Data Engineer Associate cert.
Wow — was I in for a surprise.
I failed the test in April, only made it to the 645 mark and you need at least 720. And this was an Associate level test. Even though AWS clearly states that the test is geared towards individuals with 1–2 years of AWS experience ALONG with 2–3 years of Data Engineer hands-on work experience.
While the test is not a “Pro” level exam, it is definitely above the Associate level. This AWS certification is much, MUCH more difficult than the GCP cert. More permutations of Data Engineering, using many MORE AWS tools than what GCP uses.
NOTE: A Question — — I am always pondering AND VERY curious as to how many folks have easily passed the AWS Data Engineer exam & gained the certification….???
I back-tracked and reviewed most of my studying various courses and a couple hundred pages of notes I put together while studying and taking multiple course tests.
Over the next two (2) months, I would set up the next exam only to reschedule it for later because I did not believe I had enough studying under my belt.
Told my wife, that this test date was it, taking it — even though I believed I STILL needed one more week (again).
The test was successful — even if it took over 10 hours for the results to be emailed to me. Previously in all other results, it would only take 2–3 hours to get the results.
- End result — AWS Certified Data Engineer Associate…. AND
- GCP Pro Data Engineer certified…
More studying ahead though.
In late 2024, I will return to continue work on obtaining an ML/AI engineer certification…. While I am working full time — this time.
For the past months, we were living on my savings and chewing through it….
But it is highly interesting that I am not getting many Commercial/Private company bites, even though I explicitly stated that I am open to those roles as well. Kinda want to return to the commercial world and give up security clearances and polygraphs. Limitations such as not able to:
- take medical marijuana without going through MAJOR hurdles
- — — — since any marijuana usage is an evil, nefarious, sinister thing to do — as initially instigated decades ago in the federal government,
- bring in ANY kind of electronic devices because you might be perceived as a spy or thief
- — — — and people with hearing aids or other medical devices have to jump through hoops as well — especially for wheel chairs, knee braces or metallic knee replacement parts, etc.
Now, for those who are curious about what I did to gain the AWS Data Engineer cert, I studied, big time.
Multiple courses, from:
- CloudAcademy — a full AWS Data Engineer course, with labs, quizzes and a full exam — course is 54 hours long… updated, not every week but they do update the course (which means if you come back to it in a month, your completion rate will drop from 100% to however much they have updated various lessons/lectures/labs/quizzes that you’d need to re-take — this is a good thing)
- ACloudGuru — at this time, they only have three courses, one for each domain but not all four domains — and no full set of exam questions to test yourself with — they do updates as well
- Whizlabs — full AWS Certified Data Engineer Associate Certification course and exam — over 26 hours — 280 Questions, quizzes, 120 Videos, 35 Labs (this is also available now as a standalone course)
For the three courses above, I have annual memberships for them. For the Udemy courses below, I decided not to do the annual but pay for individual courses.
Udemy (periodically updated and good for lifetime…)
- Thomas Haas — BEST test questions I’ve seen — a set of questions for each of the four domains and two full tests (65 & 54 questions) — BEST is used because he does A DAMN good job of explaining correct and incorrect answers you chose — good job of updating the exams periodically, a very good thing
- Maarek / Kane — Sundog — their 22 hours long course (AWS Certified Data Engineer Associate 2024 — Hands On!) is okay and a decent full exam but not the best answer explanations (not to me anyway)
- Maarek / Kane — Sundog — they have a course (AWS Certified Data Engineer — 3 Extra Practice Exams!) with three full exam tests — the explanations are better but Haas has them clearly beat
- Maruchin Tech — you can try his 9 hours long course (AWS Certified Data Engineer Associate (DEA-C01) Training) if willing — seemed okay
YouTube — multiple videos… multiple videos watched
So, I did all the above and when I back-tracked, I re-did CloudAcademy — just re-did each course and each lab. Along with Haas’ exams and Sundog’s exams.
But I also paid for a month of AWS Skill Builder for $29 (per month — CANCEL it after you are satisfied and passed the test)
- Exam Prep Enhanced Course: AWS Certified Data Engineer Associate (DEA-C01). The course goes over Data Analytics (2 parts), all four domains (each having 5 quiz questions) and then there is the full test exam at the end — BEST test with solid answer explanations… when you go into review mode.
- also, for each domain there is a very decent lab
- Also be sure to take advantage of AWS’ FREE 20 question test — you can retake it as many times as you’d like. Exam Prep Official Practice Question Set: AWS Certified Data Engineer Associate (DEA-C01)
For GCP, I did ~40% of the 56 hours of video course from Udemy — Data Engineering Essentials using SQL, Python, and PySpark from ITVersity and CTO at Analytiqs, Inc — just the essential parts I believed I needed last year.
- This is a very good course, just have to deal with the accent and speech mannerism. They go into detail for GCP, SQL and Python as well as PySpark which I have not made time for yet.
Here is the fun part. If you are anything at all like me, you like to dig and research. When the course is 54 or 56 hours long — that time can be doubled or tripled because you stop watching the video to go research what the host is talking about or you want more clarification or you want to see better or more examples of something — like me. I like to think everything through as much as possible when studying, cannot seem to help it — just my nature but it gives me a clear understanding of what is going on.
- Just be prepared to spend a LOT more time studying if you opt for an over 50 hours long course…
And remember, various instructors give different educational experiences in:
- The topics they choose to talk about
- how they discuss the topics
- how they pose the various questions you need to answer and
- the explanations for those answers
Some instructors you come across, you may or may not like their style
- that is the explicit reason to why I had multiple courses — if the instructor/writer did not do a good job, the end results were not good enough for me to retain all the course material — only specific areas that I made a VERY pointed effort to retain
Now, now I have to sit down and read ALL of the many, MANY Data Engineer PDFs I saved to read more of after passing the certification… and doing more labs… and doing more Data Engineer projects….
This has been my journey to obtaining, first, the GCP Data Engineer cert and then tacking on the AWS Data Engineer certification…
Hope you have a great journey as I did — just not as expensive for most of a year…
