Foone🏳️⚧️
if I had a lot more time I think I might write a book on my ideas about "adversarial automation".
The idea that the point of computers is to help the humans do their job faster and easier, and sometimes the computer or the software on it is the enemy in that battle.
Foone🏳️⚧️
because I see a lot of people approaching automation from this attitude of "software/sites should have APIs so that users can write software to automate it!"
and while that's not wrong, exactly, it's also not the attitude I think makes the most sense, you know?
We do not ask for access. We don't need to get permission to be able to automate our tasks.
Foone🏳️⚧️
There is always API 0: acting like a human/browser/user.
The first API is "fuck you I'm doing it anyway". Any additional API the program provides is merely a helpful shortcut
Foone🏳️⚧️
like, if your API doesn't provide me a follow_user() call, but the user can follow anyone by clicking one link?
Your lack of a follow_user() call is not going to stop me. I'm just going to click the link, automatically.
Having an API 1.0 doesn't mean API 0 goes away.
Foone🏳️⚧️
You see the point of this a lot in API design, where a company is like "okay we made an API but we limited it a bunch because we are scared about cheaters/bots/scrapers/whatever", while the things they limit are things a user clicking links can do in 2 seconds.
Foone🏳️⚧️
And I think this is an under-discussed part of automation because it's associated with spammers and such, but they're only one possible user of this. By making it better known it can get used for more legitimate uses
Foone🏳️⚧️
The basic philosophy of adversarial automation is that the software/website is the enemy.
Foone🏳️⚧️
You don't control it, so it can't be consider an ally in this automation.
Foone🏳️⚧️
I'm talking less like "you're in a constant arms race with the people maintaining the official API as they try to stop your spamming" and more like "Your lab depends on this program from 1996 and there's no updates and no way to automate it"
Foone🏳️⚧️
and the answer is really that of course you can automate it. Stick it in a VM, OCR the screens, inject your own DLLs, puppet the keyboard and mouse!
Foone🏳️⚧️
my point is that every program, every website, DOES expose an API, you just need to know how to best use that API.
That API being "the access they provide for humans"
Foone🏳️⚧️
For websites this is forms and links. For desktop applications this is buttons and windows and keyboards.
Foone🏳️⚧️
And I think (in part because it's affiliated with Bad Actors like spammers), a lot of programmers don't consider all their options in these areas.
Foone🏳️⚧️
And that's really a shame. Computers should be used to automate things. We spend way too much time dealing with shitty sites and shitty programs because we have no choice and think we can't automate them away.
Foone🏳️⚧️
well, that's wrong. We can absolutely automate them, it just takes a little more work and some different strategies
Foone🏳️⚧️
I think of this as a short term vs long term thinking sort of problem. Like, a lot of programmers are stuck in the "should" part of thinking about programs and sites.
Yes, the program SHOULD be open source, so you can just fix the UI. Yes, the website SHOULD have an extensive API so you can easily automate it.
Foone🏳️⚧️
I agree with all that!
but... it doesn't.
Foone🏳️⚧️
and if you want to automate it today, your only options are to be adversarial about it. It's the enemy, you pretend to be a human user and automate the interactions with the app/site. It's the only way.
Foone🏳️⚧️
by all means, try to switch to open source alternatives or get them to fix it or add an API.
But at the end of the day that's asking "the enemy" to do something for you, and they are under no obligation to listen to you.
(They may not even exist anymore, given that a lot of the times when I've used this sort of Adversarial Automation it's been focused on software from decades ago)
Foone🏳️⚧️
It's also a thing that intersects with the way a lot of people online are thinking about computer-use as something they do as a personal hobby, you know? They can run any OS, any software they can legally (or even illegally) install, they can use any options they want
Foone🏳️⚧️
But the fact is, often times people have jobs where they aren't self-employed and have to work for other people, and those other people can be like "you need to use FooBaz 2007 for this job".
Would it be easier to automate if you were using OpenBaz? Certainly! But your boss can still tell you "no, we're not switching to OpenBaz, we need to use FooBaz 2007"
Foone🏳️⚧️
And now your options are basically just:
1. Get a different job
2. Use FooBaz 2007 manually
3. Adversarially automate FooBaz 2007
Foone🏳️⚧️
One example where this came up in my career was when I was working for an educational book creator/publisher. Apple had just added a bookreader tool for iphones/ipads/etc, and we had a lot of colleges asking if we could provide our textbooks in that format.
Foone🏳️⚧️
well, at the time the only way you could make books for apple devices was to use the book creation program, which was basically a word processor. It was focused around the idea that you would create your books inside it.
Well, we already had our books created.
Foone🏳️⚧️
We didn't want to have someone retype them.
We could import them as plain text (or DOC, I think?) and that'd get the actual text content of our books with some minor formatting, but we had very interactive and multimedia books. Tons of images, cross links, quizzes, and so on. Pretty much all things that the apple book format supported, but didn't support importing.
Foone🏳️⚧️
So our options were basically:
1. Hire a bunch of people to painstakingly re-create our books inside the Apple Books tool
2. Adversarial Automation, baby!
Foone🏳️⚧️
We figured out how much could be imported, and what was left out. We figured out the limitations of the undocumented applescript interface. We figured out we could build complex HTML documents, copy them, and then have the keyboard automation press "cmd-V" and they'd be brought in without issues.
We automated away the bad UI that was going to make it too expensive to publish on apple platforms.
Foone🏳️⚧️
Should Apple have provided better docs and interfaces and APIs?
Yes, of course!
We asked for them.
But at the end of they day, they may not. And we need to publish this stuff soon, not in several years when Apple decides it might be a good idea for the next revision
Foone🏳️⚧️
My overall point is something like:
By all means, use APIs and official channels and built-in scripting support if you can.
But remember those are only shortcuts to automation. You can always ignore them and off-road.
Foone🏳️⚧️
My silliest example of this sort of thing:
I was automatically taking screenshots of a DS game in an emulator. my program would load a savestate, jam some new data into the DS's RAM, hit a button, then screenshot it. But the emulator was showing a "SAVE STATE LOADED!" text overlay over the game's window, no matter what option I set.
Foone🏳️⚧️
I go on the dev's discord/IRC, talk to them about making it an option, they say they've considered it but it's low priority.
I look into building the software myself, but it's very complicated on windows, with a lot of dependencies and such...
Foone🏳️⚧️
I open the EXE in a hex editor. Find the string "SAVE STATE LOADED!", and change the first character to a NUL.
Now the emulator is still showing the message, but since it's zero characters long, it's invisible. Problem solved.
Foone🏳️⚧️
my overall thesis is, I guess, that programmers need to remember they can say "fuck you" to software more often.
Bo Brinkman
@foone I have a raspi with a speaker in my garage that has recordings of my voice so I can give voice commands to a smart speaker when I'm out of the house. :)
Foone🏳️⚧️
@bo_brinkman nice!
Foone🏳️⚧️
@ianh oh definitely. accessibility is so often a last minute concern (if they thought about it at all) that it only makes sense to go "screw you" and build your own accessibility functionality.
ianh
@foone great thread. i think it's worth mentioning the connection to accessibility tools and APIs as well. often people are put into the same position as a screen scraper when faced with software whose "API 0" is particularly restrictive (e.g. requires acting like an abled user in a particular way, like having to interact with things via moving a mouse and clicking, rather than like a human in general who may want to or have to interact with the software via different modalities)
ianh
@foone and you see this connection a lot in practice -- accessibility hooks are quite useful for adversarial automation, and adversarial automation techniques can be used to build accessibility tools too
Christen Lofland
@foone You are basically describing my full time job. Just the automation is for internal systems, that are ancient and horrible. They do often complain that I’m sending them too much traffic, but that is because nearly all of the usage of their system is via my automation and the simple user interfaces I created for it.
Foone🏳️⚧️
@EkpyroticFrood very cool!
Are y'all hiring? :)
Foone🏳️⚧️
@usagi @collette yeah. Never use an api if you can just cast a spell on the software.
Maybe that'll be the title if "adversial automation" doesn't work out.
Computers for Witches.
Or "How to use computers like a witch"
Collette's Brain
@foone “I open the EXE in a hex editor.” oooo what magic is this??
Usagi
@collette@mastodon.social @foone@digipres.club A hex! :neocat_flop:
０７６萌ＳＮＳ is a social network, courtesy of ０７６. It runs on GNU social, version 2.0.2-beta0, available under the GNU Affero General Public License.
All ０７６萌ＳＮＳ content and data are available under the Creative Commons Attribution 3.0 license.