Compsci prof emits tool to weaponize Python's insecure pickle files to now hopefully get everyone's attention
(2021/03/19)
- Reference: 1616143809
- News link: https://www.theregister.co.uk/2021/03/19/python_pickle_format_insecure/
- Source link:
Evan Sultanik, a computer security researcher with Trail of Bits and an adjunct computer science professor at America's Drexel University, has unpacked the Python world's pickle data format and found it distasteful.
He is not the first to do so, and acknowledges as much, noting in a recent [1]blog post that the computer security community developed a disinclination for pickling – a binary protocol for serializing and deserializing Python object structures – several years ago.
[2]
Even Python's own documentation on [3]the pickle module admits that security is not included. It begins, "Warning: The pickle module is not secure. Only unpickle data you trust," and goes on from there.
Yet developers still use it, particularly in the Python machine learning (ML) community. Sultanik says it's easy to understand why, because pickling is built into Python and because it saves memory, simplifies model training, and makes trained ML models portable.
[4]
In addition to being part of the Python standard library, pickling is supported in Python libraries NumPy and scikit-learn, both of which are commonly used in AI-oriented data science.
According to Sultanik, ML practitioners prefer to share pre-trained pickled models rather than the data and algorithms used to train them, which can represent valuable intellectual property. Websites like PyTorch Hub have been set up to facilitate model distribution and some ML libraries incorporate APIs to automatically fetch models from GitHub.
[5]Happy birthday, Python, you're 30 years old this week: Easy to learn, and the right tool at the right time
[6]Python swallows Java to become second-most popular programming language... according to this index
[7]Facebook boffins bake robo-code converter to take the pain out of shifting between C++, Java, Python
[8]Python 2 bows out after epic transition. And there was much applause because you've all moved to version 3, right? Uh, right?
Almost a month ago in the PyTorch repo on GitHub, a developer who goes by the name [9]KOLANICH opened [10]an issue that states the problem bluntly: "Pickle is a security issue that can be used to hide backdoors. Unfortunately lots of projects keep using [the pickling methods] torch.save and torch.load ."
Other developers participating in the discussion responded that there's already a warning and pondered what's to be done.
Hoping to light a fire under the pickle apologists, Sultanik developed a tool called [11]Fickling to assist with reverse engineering, testing, and weaponizing pickle files. He hopes security engineers will use it for examining pickle files and that ML practitioners will use it to understand the risks of pickling.
Sultanik also developed a [12]proof-of-concept exploit based on the official PyTorch tutorial that can inject malicious code into an existing PyTorch model. The PoC, when loaded as a model in PyTorch, will exfiltrate all the files in the current directory to a remote server.
"This is concerning for services like Microsoft’s [13]Azure ML , which supports running user-supplied models in their cloud instances," explains Sultanik. "A malicious, 'Fickled' model could cause a denial of service, and/or achieve remote code execution in an environment that Microsoft likely assumed would be proprietary."
Sultanik said he reported his concerns to the maintainers of PyTorch and PyTorch Hub and apparently was told they'll think about adding additional warnings. And though he was informed models submitted to PyTorch Hub are "vetted for quality and utility," he observed that there's no effort to understand the people publishing models or to audit the code they upload.
Asking users to determine on their own whether code is trustworthy, Sultanik argues, is no longer sufficient given the [14]supply chain attacks that have subverted code packages in [15]PyPI , npm, RubyGems, and other package registries.
[16]
"Moving away from pickling as a form of data serialization is relatively straightforward for most frameworks and is an easy win for security," he concludes. ®
Get our [17]Tech Resources
[1] https://blog.trailofbits.com/2021/03/15/never-a-dill-moment-exploiting-machine-learning-pickle-files/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://docs.python.org/3/library/pickle.html
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] http://www.theregister.com/2021/02/20/python_at_30/
[6] http://www.theregister.com/2020/11/12/python_swallows_java_to_become/
[7] http://www.theregister.com/2020/06/12/facebook_transcoder_ai/
[8] http://www.theregister.com/2020/04/21/the_final_python_2_release/
[9] https://github.com/KOLANICH
[10] https://github.com/pytorch/pytorch/issues/52596
[11] https://github.com/trailofbits/fickling
[12] https://github.com/trailofbits/fickling/blob/master/example/pytorch_poc.py
[13] https://azure.microsoft.com/en-us/services/machine-learning/
[14] https://www.theregister.com/2021/02/10/library_dependencies_attack/
[15] https://www.theregister.com/2021/03/02/python_pypi_purges/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
He is not the first to do so, and acknowledges as much, noting in a recent [1]blog post that the computer security community developed a disinclination for pickling – a binary protocol for serializing and deserializing Python object structures – several years ago.
[2]
Even Python's own documentation on [3]the pickle module admits that security is not included. It begins, "Warning: The pickle module is not secure. Only unpickle data you trust," and goes on from there.
Yet developers still use it, particularly in the Python machine learning (ML) community. Sultanik says it's easy to understand why, because pickling is built into Python and because it saves memory, simplifies model training, and makes trained ML models portable.
[4]
In addition to being part of the Python standard library, pickling is supported in Python libraries NumPy and scikit-learn, both of which are commonly used in AI-oriented data science.
According to Sultanik, ML practitioners prefer to share pre-trained pickled models rather than the data and algorithms used to train them, which can represent valuable intellectual property. Websites like PyTorch Hub have been set up to facilitate model distribution and some ML libraries incorporate APIs to automatically fetch models from GitHub.
[5]Happy birthday, Python, you're 30 years old this week: Easy to learn, and the right tool at the right time
[6]Python swallows Java to become second-most popular programming language... according to this index
[7]Facebook boffins bake robo-code converter to take the pain out of shifting between C++, Java, Python
[8]Python 2 bows out after epic transition. And there was much applause because you've all moved to version 3, right? Uh, right?
Almost a month ago in the PyTorch repo on GitHub, a developer who goes by the name [9]KOLANICH opened [10]an issue that states the problem bluntly: "Pickle is a security issue that can be used to hide backdoors. Unfortunately lots of projects keep using [the pickling methods] torch.save and torch.load ."
Other developers participating in the discussion responded that there's already a warning and pondered what's to be done.
Hoping to light a fire under the pickle apologists, Sultanik developed a tool called [11]Fickling to assist with reverse engineering, testing, and weaponizing pickle files. He hopes security engineers will use it for examining pickle files and that ML practitioners will use it to understand the risks of pickling.
Sultanik also developed a [12]proof-of-concept exploit based on the official PyTorch tutorial that can inject malicious code into an existing PyTorch model. The PoC, when loaded as a model in PyTorch, will exfiltrate all the files in the current directory to a remote server.
"This is concerning for services like Microsoft’s [13]Azure ML , which supports running user-supplied models in their cloud instances," explains Sultanik. "A malicious, 'Fickled' model could cause a denial of service, and/or achieve remote code execution in an environment that Microsoft likely assumed would be proprietary."
Sultanik said he reported his concerns to the maintainers of PyTorch and PyTorch Hub and apparently was told they'll think about adding additional warnings. And though he was informed models submitted to PyTorch Hub are "vetted for quality and utility," he observed that there's no effort to understand the people publishing models or to audit the code they upload.
Asking users to determine on their own whether code is trustworthy, Sultanik argues, is no longer sufficient given the [14]supply chain attacks that have subverted code packages in [15]PyPI , npm, RubyGems, and other package registries.
[16]
"Moving away from pickling as a form of data serialization is relatively straightforward for most frameworks and is an easy win for security," he concludes. ®
Get our [17]Tech Resources
[1] https://blog.trailofbits.com/2021/03/15/never-a-dill-moment-exploiting-machine-learning-pickle-files/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://docs.python.org/3/library/pickle.html
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] http://www.theregister.com/2021/02/20/python_at_30/
[6] http://www.theregister.com/2020/11/12/python_swallows_java_to_become/
[7] http://www.theregister.com/2020/06/12/facebook_transcoder_ai/
[8] http://www.theregister.com/2020/04/21/the_final_python_2_release/
[9] https://github.com/KOLANICH
[10] https://github.com/pytorch/pytorch/issues/52596
[11] https://github.com/trailofbits/fickling
[12] https://github.com/trailofbits/fickling/blob/master/example/pytorch_poc.py
[13] https://azure.microsoft.com/en-us/services/machine-learning/
[14] https://www.theregister.com/2021/02/10/library_dependencies_attack/
[15] https://www.theregister.com/2021/03/02/python_pypi_purges/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YFSEUhisDlmbmjDIiCVcowAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
"they'll think about adding additional warnings"
Pascal Monett
Oh, that's just great.
There's already a warning nobody's paying any attention to. The solution is obviously to add another warning.
Would the maintainers of Python be bitten by the worm of administrative thinking ? You need to lock that functionality down. You need to ensure that pickled files are encrypted or something. You need to bake security into it somehow. I don't know, I don't have the answer, but just slapping another warning on and calling it a day is not the solution.
Perfect.
Pleasing.
Perfunctory potable?