As we reported, the ATF seems to be singling out the activities of Austin’s Silencer Shop as one of the culprits for their e-Forms website being… well… terrible. We posted the email earlier this afternoon where they outline a new reboot schedule that takes the system offline for nearly half the day, and now we have confirmation of an additional attachment that was apparently sent only to SOT/FFL holders and not to Form 1 registered users (like me). The gist of the letter is that the batch uploads were breaking the system, and after reviewing the letter Silencer Shop says their methods are now in compliance and they can keep rolling with the batch uploads.
URGENT NOTICE TO: All Registered ATF eForms users
As you are aware for several weeks the eForms application has been experiencing some technical problems that have rendered the application difficult or impossible to use. We apologize for the issues that our customers have been experiencing. We have been diligently working to resolve these issues and have sought the assistance of IT systems experts.
Those experts have advised us of one major problem; that the eForms system is NOT designed to accept batch uploads. We have noted that several of our registered eForms users have designed and use a “batch/multiple submission process” that we have determined has been contributing to the problems that eForms is experiencing. ATF did NOT grant these users any authorization to use this process. eForms was designed to work as a serial application, with each user making one submission at a time. The “batch/multiple submission processes” are creating a situation where processes are initiated and not completed within the eForms application. This is causing “memory allocation errors” within the eForms application. The result is that many users are prevented from being able to log-in. Another symptom is that once users log-in, the system moves so slowly that it is taking 20 minutes or more to move from one screen to the next. Users are also reporting that they are unable to successfully submit an application even if they are fortunate enough to make it to the submission page.
Because the “batch/multiple submission processes” have been identified as a major contributing factor to these problems, we are requesting that any registered eForms user who is currently using any “batch/multiple submission process” discontinue the use of any such process IMMEDIATELY. Any user who continues to use a “batch/multiple submission process” for eForms, after the dissemination of this message will have his/her access to the eForms system suspended or revoked. Please understand that we are not asserting that the users of the “batch/multiple submission process” did anything wrong. Unfortunately, our system was not designed to handle batch processes which are causing unexpected problems for our system.
ATF is currently assessing whether there are other improvements that can address the problems identified. One of the issues that we will address in the future will be the creation of a batch process that will be available to all eForms users without a detrimental effect on the performance of the eForms system. Until that time we request your compliance with the above, and appreciate your cooperation and patience.
If I’m reading this right, then the real problem isn’t the batch uploading of forms to the website — the problem is people submitting multiple forms at the same time. Whether that means an automated system doing it in the background or an FFL having all of their employees manually entering multiple customer’s information during the day, it’s the same effect.
While this may appear to be an effort to stop Silencer Shop’s batch uploading of forms, in reality this is really advice that every FFL using the system needs to heed. One form at a time per account, no matter if you’re using different computers, is the way to keep the system from crashing.
The simplest solution would be to remove silencers from the NFA. But this, after all, is the federal government, where “simple” and “efficient” are a foreign language at best and curse words at worst.
SBR/SBS’s can go too.
Why not the whole thing?
Silencers aren’t even weapons, they’re PPE. How is it even covered by the national Firearms act? Honestly, hope is sitting pretty low right now. The news is cheering a net 900k people (6 million already had it but were dropped) who were previously uninsured for the low low price of a 600 million for the web site plus 2 trillion or so.
Imagine that a government website experiencing technical difficulties. Nah, never happens.
The only reason the ATF is having trouble is because of the computer they deployed to handle the eForms — a 20 year old computer running Windows 3.1 in the corner of their basement.
No sh1t, exactly what I’m saying.
Hey I responded to your post on the Florida thing, about debating Liberals.
Take a look I think you may find it interesting.
Pffft. . . you know they took the old ENIAC out of mothballs for the job, don’t lie.
I haven’t worked in an IT shop that needed to reboot servers on a regular basis for the past 15 years.
Pathetic.
Installing updates is pretty much the only reason to reboot these days.
not if you have finally freed yourself from Microsoft and moved onto Linux! 😀
For userland stuff, right on. Kernel and other nonuserland stuff?
well ok , I was just celebrating being off windows. 😉
Roger that, but you made one fatal mistake. FreeBSD rocks! 😀
Meh. The ratio is a lot better than it used to be. My Windows boxes reboot from patches at about 1.5 times the rate of my Ubuntu laptop reboots from patches. They’re close enough to my needs that the difference isn’t relevant.
Of course, my reboots don’t take half the day. This ATF system is crap.
” This is causing “memory allocation errors” within the eForms application. ”
Good Lord, this is truly sad. That ooor GC thread is just spinning because disposables aren’t being cleaned up.
It was not designed to do batch because crappy developers have done such a poor job memory reclaimation cannot occur so they have to bounce the hosts.
As a taxpayer, I want my money back. This is inexcusable.
I can make one phone call and spin up hundreds of virtual servers that can scale up or down as horsepower is needed. Depending on the code and what is needed I am thinking it should not take a ton of time or money to spin up a temporary fix while they work on a long term solution.
Not to be an ass, but who would you call?
Remember, the ATF has access to your information that they cannot place on another company’s systems. By law. They HAVE to protect it. That means you can call up Amazon and get a dozen servers in 30 minutes… but you can’t put any data on it. Did you put out a bid for the server contract? Did you make sure you gave preference to a minority-owned business? Did you happen to have an authorized contract for emergency computer services?
Government IT – especially at the federal level – is chock full of extra roadblocks like this. It’s easy to say “Just build a system using technologies X, Y & Z, hire industry leader F and call it a day!” but the gov’t doesn’t work like that. When they do work like that, they get sued and you get your department on the front page of the NYT talking about “no bid contracts” and “directed the contract to the companies of political donors” and all that jazz.
The FAA set the precedent.
http://blog.tedroche.com/2004/10/08/windows-server-at-faa-crashes-every-497-days-unless-maintained-rebooted/
So Silencer Shop, submitting all their forms early in the morning once per day, is causing this problem? That makes no sense. After Silencer Shop submits their forms and the servers undergo a reboot, there shouldn’t be any lingering problems, yet the servers require reboots several more times the same day.
The real problem is load. Load load load. The morons who cooked up eForms made no professional attempts to test for load – lots of people logging into and submitting eForms throughout the day – so their WTF-tier system is bogged down all day long. In fact, if NO ONE ELSE ever logged into eForms and only Silencer Shop submitted forms for all the transfers in the country, you can bet performance would improve overall because they’ve figured out a way to submit forms efficiently without making redundant service queries.
Sucks for those who waited to buy…..enjoy the extra wait of singular entries…..the last 2 suppressors I bought were batch loaded by SS 2 weeks ago. HA!
I’m actually an IT professional and can actually tell you from the software that our programmers write and software developed by other companies, this is just the govt failing yet again. It isn’t difficult to write a program (or in this case a web page/eForm) that can handle multiple submissions at the same time. Once again, the government fails to do something right…part of the reason I carry…God only knows what they will screw up next.
Yep. IT pro, 25+ years. I hear you.
Been doing the gov’t IT development for the past 15 years or so. I’ve seen way too much of this kind of system development.
On a positive note, not every gov’t system is like this. But it does happen way too often.
+1. This whole part:
“eForms was designed to work as a serial application, with each user making one submission at a time. The “batch/multiple submission processes” are creating a situation where processes are initiated and not completed within the eForms application. This is causing “memory allocation errors” within the eForms application. The result is that many users are prevented from being able to log-in. Another symptom is that once users log-in, the system moves so slowly that it is taking 20 minutes or more to move from one screen to the next. Users are also reporting that they are unable to successfully submit an application even if they are fortunate enough to make it to the submission page.”
Just reeks of incompetence.
To add insult to the injury, these people are getting $200 for every transfer. Imagine that you’re working for a business writing server software where every client hit is $200 of profit to your company. If you end up writing something that can’t handle concurrent requests because of “memory allocation errors”, how long do you think you’d keep working there? Geez.
Being in IT myself, I have no problem with their explanation of serial/batch processing. Many times I have seen poor planning in a project lead to poor performance when the app is scaled up. However, I do not understand why batch uploading worked at all if it wasn’t designed for that. My guess is it was designed to do batch processing but it doesn’t work well. If they wanted batch to work, it would work.
The Feds now have a entire infrastructure dedicated to processing applications for the ACA that, as of this morning, is no longer needed. It doesn’t work well with millions of users, but should be sufficient for the thousands of applications for NFA devices -especially from a set of “SOT/FFL holders” they already know about. I suspect it too was not designed for batch input. FWIW, the eForm and ACA system are certainly not compatible or the ACA infrastructure would be used to speed up the eForms system.
Being in IT myself, I have no problem with their explanation of serial/batch processing. Many times I have seen poor planning in a project lead to poor performance when the app is scaled up. However, I do not understand why batch uploading worked at all if it wasn’t designed for that.
I can see why it might work sometimes / for a while, but need an every-5-hours reboot to keep working: memory leaks in the app (as they suggested in the letter above).
Dumbass developer(s) didn’t free up memory they’d dynamically allocated. Get your asses back to “first semester software development” (or whatever the equivalent is in the book learnin’ world).
>> However, I do not understand why batch uploading worked at all if it wasn’t designed for that.
What it means is that they’re misusing the term “batch” here. What was almost certainly actually happening is that the guys wrote their own “batch submission” system that just rapidly submits prefilled formed over HTTP, in exact same way as users do manually, just as a fast burst at a certain convenient moment in time.
Got it.
The more interesting thing here is that the ATF views FFL holders as “customers.” Made me LOL.
Memory allocation errors do not happen because people are submitting batch forms. They happen because of half- baked programming.
I’m a “cloud computing” guy in Silicon Valley. We in the industry have learned how to make systems scale to accommodate arbitrarily large loads – consider Amazon and Google and Intuit (TurboTax) to name just a few who use very different approaches under the hood. To handle their piddling load, ATF needs to catch up to the turn of the millennium or so.
Bob….so where does one go to find…or how does one who has… what they believe to be a great computing / IT idea…(or ideas)….how does one get their ideas into Silicon Valley…or at least get someone there to listen to your concept? I’m a retired US DoD Executive…I have zero education in the development of IT related products…but I have a decades of international business and marketing experience…? Would you have a society or guild that someone can contact….? My concepts have huge potential, or I wouldn’t waste the time asking…
Regards,
RJ O’Guillory
Author-
Webster Groves – The Life of an Insane Family
1. Business Plan
2. Venture Capital
3. Hire competent contractors
4. Demo Beta Product
5. Get bought out
6. Blow fortune on NFA and ammo
Gene..Ha! Thanks….Can I just jump to step 5? The information I left out of my prior message…is that I was diagnosed as being genetically epileptic and suffer from really bad seizures…which steal my memory…both short & long term….and in 2010…after I’d given up driving..I had a seizure…and drove a car 7 miles through the mountains…only to drive off a 200 foot cliff at 70mph…ripped a 19 staple hole in my head…broke my back and neck in multiple spots…a few broken ribs and smashed tailbone…so my business dependability is almost nil….but my creativity levels are still high….so to speak…and I know the international customer pretty well…and the law of big-numbers…
Thanks for the feedback…
RJ
On the other side of the coin, how do you think customers will feel about the additional $100 processing fee charged by some of these companies for something the FFL dealer can do for them for free. I have purchased a suppressor in the past and would not want to line someone’s pockets when I can get it for nothing. That’s all their doing. It was good while it lasted, just submit them like everyone else. Why should they be better than anyone else at my expense.
who is charging $100 processing fee? Just curious.
A serial application only? For god’s sake it’s 2014! Even cell phone apps can do more than one thing at a time!
A memory allocation error is a problem with the website, not the user. The user is using the system the way that it was supposedly built to be used. It is a classic example of incredibly stupid design on the part of the engineers who built the system.
Usually the systems designed to take our money work pretty well. IRS has much bigger scale than this and concentrated into tax season.
Customer wants to pay $200 to get a form processed. It is 2014 and IT is not black magic. How bad do you have to eff up to get fired from a gubmint job?
PS – you can throw a lot of hardware at a crap system to little effect, although it sounds like under powered HW is part of the issue.
I dunno. Even under powered hardware should be able to handle a hundred form submissions per minute. This strikes me as a bad architecture, single-threaded process design and a memory leak somewhere.
My partner and I absolutely love your blog and find many of your post’s to be precisely what I’m looking for. can you offer guest writers to write content to suit your needs? I wouldn’t mind creating a post or elaborating on some of the subjects you write with regards to here. Again, awesome blog!
Your style is unique compared to other folks I have read stuff from. Thank you for posting when you have the opportunity, Guess I’ll just book mark this blog.
What’s up, the whole thing is going well here
and ofcourse every one is sharing facts, that’s truly fine, keep up writing.
Thanks for finally talking about >UPDATE: ATF Issues Letter About Multiple Uploads, Silencer Shop
Already In Compliance – The Truth About Guns <Liked it!