By Christopher Cicchitelli
As a fervent Second Amendment supporter and the founder of a technology company, I’ve followed the ATF eForms story with great personal interest. It showed great promise at first, cutting wait times down to under 60 days…and then it imploded. Exactly what happened was a question we’ve all asked, and until reading the ATFs announcement yesterday I didn’t really have a good idea. However, in that announcement I believe the ATF tipped us off to the key that unravels the mystery . . .
First, some full disclosure: prior to starting my company, CastleOS, I worked as a contractor to a government R&D lab where, among other responsibilities, I designed integrated systems and managed servers for multiple civilian and military agencies.
Now for some key facts we know: despite the ATFs initial claims otherwise, the eForms system was unusable even when batch processing was not taking place. Even if we presume Silencer Shop initiated batch processes outside the 4-5 AM window they claim, I find it hard to believe they did it all day every day.
In addition, in today’s ATF announcement, they claimed there are “memory allocation errors” within the system. In order to keep it usable for all, it needs to be rebooted multiple times a day, taking up to an hour for each reboot. That’s key, and I’ll come back to it later.
In addition to these facts, there are several questions we are all asking. How was batch processing ongoing if the system wasn’t built for it? Why does the system need to be rebooted every few hours (this is 2014 after all!)? Whose fault is it really?
After doing some research, I think I have narrowed down the realm of possibilities for the answers to those questions, so here goes. One of the first things I noticed when visiting the eForms website (well, after I got to the website to load – it took a few minutes) was the technology it’s built upon: JavaServer Pages. To say JSP isn’t exactly the most popular platform these days would be an understatement. While still around for legacy reasons, you don’t often seen it used for new projects without a specific reason – a reason I don’t see the ATF having. From my own experience as a contractor, I wonder if this was a matter of the contracting company having available labor in this speciality, and thus pushing the platform on the ATF. If so, the ATF contracting officer would certainly deserve some blame for this as well.
The other immediate observation was the speed of the website loading up. It repeatedly took about five minutes to reach the login screen at around 7pm EST. What this shows is a faulty system architecture design. A proper design would separate all tiers, including at the servers themselves, into the following three categories: user interface, data broker, and data warehouse.
There’s no doubt that there is a heavy amount of processing of data going on in this system, and it’s slowing the whole system down – so there isn’t a proper separation of tiers in this design. The fact that the data warehousing can slow down the most basic parts of the user interface – the website itself and login page – is a major failure. So much so that for the millions of dollars spent on this website, I’d argue the government would have recourse to recoup (at least some of) the money it laid out.
Next is the question of how batch processing was happening if it wasn’t designed for it. The ATF has continually sounded the horn about the batch processing, but the truth is it’s misdirection. Whether the eForms system has a batch option or whether the Silencer Shop is using a custom method — possibly something as simple as a macro — to automate the process is 100% irrelevant. The reason is for all intents and purposes, a batch process just simulates the effect of multiple stores logging in at the same time. What we are seeing is a system that goes down with very little load, possible just a few dozen simultaneous users, or one user uploading a few dozen forms a night, and I think the reason lies in the ATF’s “memory allocation error”.
Traditionally, a memory allocation error is when data is corrupted or misplaced on actual memory. I don’t think the ATF has an actual error with the server memory itself – using JavaServer Pages should prevent that – but rather is PR spin for a more complicated resource allocation error.
What I think is happening is as forms are entered into the system, they enter a bureaucratically-inflated workflow. In that workflow, the system is probably generating loads of actual documents in addition to lots of database entries and so forth. (After all, do you really expect the ATF not to keep a backup paper trail? Pfff.) In other words, each application isn’t as simple as the typical web form we are used to using, and it does require some real horsepower to process it from start to finish.
Now normally that’s not an issue, each server has its maximum number of users, and you add servers as needed. The government even has a gov-only cloud it can use to deploy servers on demand – so keeping up with demand shouldn’t require more than a few minutes to boot up a new instance. However with the eForms system, the opposite seems to be true, and they can never bring enough servers online to keep up with demand (they tried at the beginning). I believe the reason is because load isn’t the issue, batch or otherwise, but a fundamental flaw in the design of the system: applications are gobbling up resources as they are processed, and then when complete, not releasing those resources.
The proof of that is the reboots every few hours to clear the “memory allocation errors” – if the code isn’t releasing the resource, you need to reboot. (Why it takes an hour isn’t clear – but that may have more to do with their attempts jury rig fixes into the system, and/or having to bring systems online in a certain order.)
I also believe that in the beginning, before too many people were using the eForms system, this same flaw was present, but was being covered up by nightly reboots. Once the load increased, the problem literally spread exponentially, and now the system can only run for roughly 4 hours until all the servers run out of resources.
The saddest thing about this for me, is not that the system failed, I’ve seen that far too many times before in government. It’s that so many millions were spent to bring a system online with such a fundamental flaw. I’d give the Obamacare website team a break long before I’d cut ATF any slack for this fiasco.
I remember as a 6th grader in the early 1980’s having classmates bring rifles & shotguns to school and explain how they worked for show & tell.
In New Zealand in the early 1980s, everything was run by our Prime Minister, Rob Muldoon, an effective but elected dictator. I once saw him on the hustings react to a heckler. He walked over to the guy and punched him in the gut. He ruled with a rod of iron, any journalist who criticized him was banned from press conferences. All major infrastructure was owned by the State, prices and wages were fixed by Government, and Rob ruled through fear. After 1984, this regime was overthrown, public works were privatized, and suddenly a very few people got incredibly rich. Our previously egalitarian society was gone, and has never returned. Now those who attend the right schools and know the right people have a golden path through life, and our numbers of millionaires has risen. But those of us from the wrong side of the tracks have steadily fallen into increasing poverty, and our present Government is of the rich, by the rich and for the rich. If we didn’t have guns it would be even worse. There are forces wanting complete disarmament. But thankfully deer and duck hunters make that impossible.
I grew up in Sparks, Nevada in the 80s. My mother’s boyfriend was a truck driver, and around ’83 after one of his runs he brought home a couple air force practice bombs, they looked like the kind that you would see falling out of WWII bombers in old video clips. They were probably 4 feet long and weighed 30 pounds or so. We just left them laying around in the yard.
In second grade I brought one to show-and-tell. A bomb. To my elementary school. There was no lockdown, no media circus, no suspension, no nothing. In fact it was well received. I had a hell of time carrying it to the two blocks home, being a 7 year-old. As I was slowly dragging it home I recall an OD Green jeep driving by with a couple GIs in the open cab. One of them said, “That kid has a bomb!” and they both laughed and drove on. That was all the negative attention I got.
Times have changed.
There are many gun owners who are also software developers — we should create a system and donate it to the ATF. It would be to our benefit as well as theirs.
Things were much better before America went insane and the inmates took control of the asylum.
It always mystifies me when people complain that there is no commonality of parts with the AK. I understand that the two fire the same caliber, but they are totally different weapons.
It would be like complaining that my XD does not have interchangeable parts with my Glock, or my SIG, or my Caracal.
I like the vz.58 because it is better in many ways than an AK. If it hand interchangeable parts, it would be an AK. It isn’t an AK, it is better.
The magazines have advantages over AK mags. I would not complain about the fact that my ARs don’t accept AK mags. I don’t complain that K31s don’t use the same mags as HK G3s, which don’t use the same mags as a .308 AR or an FAL. In fact, I bought a .308 AR lower once that used HK G3 mags. I sold it because of the disadvantages.
People buy weapons and the proper magazines for them, and no one complains. Except in the case of the vz.58; then they whine about the fact that it does not use the magazines from a totally different weapon and miss out on owning one of the coolest little 7.62×39 carbines ever made.
Looks like a nice package. I would love to find one of these rifles around. I have found the the old combloc guns are a lot of fun to modify. I’ve finished two Saigas and will move on to my Saiga 12 after my ARs. A Vz58 would be cool because it is different. Although I have three AR projects to finish before I really can buy another rifle.
They sounded pretty damn identical to me. So I say go for the unobstructed sight picture.
I appears to be built using the exact same platform as the online payroll app for Paychex. The odd look & feel, user unfriendly “steps” to progress through and the general overall process is identical. I think it is an off-the-shelf platform of some type. myapps.paychex.com does actually work reliably though.
People have tried to run me over with cars, TWICE.
One death confirmed. “Shelter in place.”
I wonder how this is gonna be spun.
A military base is the LAST place a shooter should think to try his luck. But, unfortunately…
I was stationed at Ft. Hood in the late 90s. It’s difficult to describe how big the post it, but it’s huge. Arms are kept under lock and key in the arms room, and take a while to access. Ammunition is stored separately, and is issued separately, and accounted for.
I’m not saying it’s right, just that it’s how things are run. Trusting thousands of Private Snuffys with live ammo in a garrison environment is going to lead to some Very Bad Things. That said, issuing every E6 and higher a sidearm would probably go over pretty danged well.
So, let’s keep an eye out for these things:
AR15 used?
Children dead?
High-cap magazines?
If none of these are present, then this will fade from the media in 2 days, tops.
Y’all need to get some wind muffs for your cameras. Considering the amount of video Leghorn shoots, it would be a worthy investment.
Again, the so-called “authorities” are directly told, in detail, to an immediate, lethal threat to the citizenry.
Again, the so-called “authorities” either deliberately ignore the warning or cut “their man” loose.
Again, the so-called “authorities” thought their fickle, impotent laws would protect them from madmen.
Again, they were wrong.
You should have compared the Silencerco Octane to the above models. I’m curious how it would perform again the osprey and the TiRant, but has a major advantage stainless steel baffles. My friend has the TiRant 45 and cleaning the baffles is a HUGE rectal irritation.
First reports: 8 wounded, 4 critically…
And they were warned that some jerk was hell-bent on doing this? Embarrassing.
They sure did name him quick.
>> The pistol grip that comes standard on the Vz. 58 is your typical combloc affair: it was made for your average 5’6’’ underfed Warsaw Pact conscript.
Can you please stop regurgitating this ridiculous myth? The reason for shorter stocks and smaller pistol grips on Warsaw Pact weapons is to make them work better with thick winter coats and gloves, not because the conscripts were “underfed”.
wait for the potis spin that more gun control is needed. sickening. And that military leaders do not allow the military to defend themselves or others on military bases. criminal.
POTUS: Get to the bottom of what happened by re-arming soldiers on military bases so they can defend themselves. Do not allow U.S. military bases to be “no gun” zones, which put all soldiers at risk of injury or death by the hands of a mentally unsound person or Muslim jihadist.
This was NOT a random act of violence, or terrorism, or whatever they want to call it. This was purely intentional to advance the gun control agenda. And YES I am a conspiracy theorist!!
when will they learn that gun-Free zones kill the innocent? all who Disarmed our troops need to face treason and murder charges
Just take a look at all the chef shows on TV. Shows called Chef Wars, Last Chef Standing, and my favorite, Knife Fights, all embrace a distinctly violent overtone for a chef/cooking show. But they want the gun wearing public to listen to their call for their customers to disarm?
“It’s a bit strange to me that you think you need to carry a gun when you’re having a cheeseburger.” Chef Brock
I think someone who was eating a burger at the Luby’s in Killean TX, would have a few words for this chef. They could have used a concealed gun on their person when George Hennard went on his shooting spree. 24 dead because Texas law said they couldn’t take their gun with them into the restaurant.
now now my good fellows i no Ray very well and he is a nice guy he sold my sisters guns and me one all so. now thats get to the bottom of this . that women cant help it she just had a house fall on her and her ruby slippers were stolen along with here magic musrooms she brought from drug riddin New York, so cut here some slack( around the neck would ) and keep an i you you little dog toto if she is flying around
I’m kind of surprised they did not declare “VICTORY!” as it would make more sense than when they DO say it.
So, you win this round, MAIG/MDA…Gun Free Zones wins more lives for your cause.
Remainder of Comment Self Moderated.
Interestingly, all of this seems to be another case where things have slowly gone left of center over time. There was a time back before the Reagan years, that soldiers, sailors, airmen, and marines often had weapons on post. A good friend of mine, who was a bachelor-officer at the time, reported to his ship on the West Coast (California, no less) with several shotguns, multiple revolvers, and a rifle or two. None of the senior officers objected and nobody was in the least bit alarmed or upset. The only comment he got was, “congratulations, Mr. Johnson. You’ve managed to increase the size of the armory by three-hundred percent…” My dad would routinely return from TDY with pistols that he’d bought wherever he’d been at, and, again, nobody batted an eye.
I, too, applaud Sgt Hayes for continuing to write articles for TTAG. Taken in context, Sgt. Hayes’ observations and conclusions illustrate how some PD’s work with the Communities they serve to avert WACO’s and Ruby Ridges’ and minimize the occurrence of needless violence between LEO’s and Citizens. We need to support this kind of police work, not condemn it.
For a long time I have believed the most egregious incidents happen in the hands of Federal Agencies and that some ‘big city” PD’s have mistakenly adopted policies that mirror the Feds out of either political agendas or a misguided sort of “penis envy” originating with the politicians who command them.
Yes, there will be more WACO and Ruby Ridge incidents, because there are isolated persons out there who will choose to initiate violence with the so-called “Authorities” and garner the attention of the Feds. Nowadays, the Feds, regard any resistance to their sense of social order (and political agenda) as some sort of “terrorist” activity which must be crushed ruthlessly.
Given the number of people in the population, the social stresses of joblessness, an increasingly impoverished “middle class”, political divisiveness, the health care disaster, educational malfeasance, moral deterioration, and overall “stupiding-down” of the average citizenry ( a short list of what’s killing the USA), there will never be a time when we can do without formalized law enforcement agencies at the local level. So. our best course of action is to consider, and act upon, ways we can make this work better for Citizens and Citizen LEO’s. I think this is what Sgt. Hayes is trying to illustrate and advocate, as I read it, and I agree with him.
If some one wants expert view on the topic of blogging afterward i recommend him/her to pay a
visit this website, Keep up the pleasant job.