Claude Application tool
See my Code: Autofiller
As I continue to familiarize myself with code writing, what I am really chasing is a deeper and more intuitive understanding of how code functions and how I can learn to view a line of code or a fragment and understand what it is directing. Code writing is increasingly less important for the individual, but understanding what is being written by an agent or GenAI is growing in importance; especially as the gap widens between those who do understand and those who allow AI to blindly type everything for them.
As I search for the best possible fit for a summer 27 internships I find myself spending a significant amount of time writing out the monotonous informative fields that should be autofilled from a resume. I feel that this cuts into my ability and effort for company specific questions (why XX industry? why NYC? why…) so instead of using a canned tool, I wanted to try and utilize ai to make a custom tool for my needs. Everything was built into Python via Spyder6.
I didn’t blindly type into the prompt “make me a good auto filler tool for applications” but rather I did some research beforehand and went into the code building process with some ideas which ended up paying off enormously as I was able to have a back and forth conversation with Claude instead of praying it knew what it was doing
My plan was essentially this: find a prebuilt microsoft tool that can capture the screen automatically, and then parse the screenshot, determine what fields we see, and input the pertinent information via a simple command to paste and tab. The code would already be populated with my information so all it has to do is paste the info related to “name” or “email”
Claude ended up suggesting this situation using the tool UI Automation, a premade tool that sees all the information on a screen, not as pixels, but as trees of objects that have properties. This is excellent, because it does away with any possible errors regarding popup notifications covering a field, cursor being in the way of spelling, or any other issue I haven’t thought of. UI Auto also determines which field I have currently selected and can find its exact AutomationID so there is no chance of guesswork in variations. (A potential issue this mitigates is being able to handle “quirky” fields like “first & last” or “Forename & Surname” instead of “name” which would be difficult for a program that only looks for a handful of keywords)
There are also a few libraries that were imported, each very powerful: Pyautogui, Pyperclip, Pytesseract & PIL.
Pyautogui allows for control over the mouse and keyboard and pyperclip can read and write the clipboard. These two are instrumental in assuming control over my role as an application filler; the clipboard is needed to literally take down my data and get into the application. Pytesseract and PIL act in this code as a failsafe, in the unlikely event that a clear AutomationID is NOT present, the code essentially sees a dark spot and has no idea what it is asking for. Also, if one of these AutomationIDs is missing there are probably many missing IDs in that page and thus the program would be useless.
PIL grabs an image, depending on the selected field, converts to grayscale, scales by a factor of two and adds contrast. Tesseract then reads the text and formats it as a string corresponding to one of the prefilled fields with my info like name or email. After this is complete it hands off the process to the clipboards tools and they fill in the info, tab, and then repeat.
After a field is found, it is crosschecked with the library of keywords; fname, first, etc are hardwired to correspond to name; it then pastes the corresponding information related to the verified field, and tabs; repeating the process from start
Coding is never going away, it builds the systems that engineers rely on and has made the foundation for the world of tech; but spending years mastering writing code can be nearly as effective as some research and a powerful AI tool. This project would have taught me absolutely nothing if i merely commanded Claude to spit out code and hoped for the best, but as with any technical project, using the resources available is never a bad idea as long as some meaningful effort is taken to understand what is happening inside.
