gsutil TypeError: cannot pickle '_io.TextIOWrapper' object

    

By: Jason Alvarez,
Follow me on twitter @0xBanana


If you’re trying to copy files or otherwise interact with google cloud storage buckets using gsutil and end up having an error similar to this one, there is a solution!

If you’re trying to copy files or otherwise interact with google cloud storage buckets using gsutil and end up having an error similar to the one below, there is a solution!

~ gsutil -m cp *.avi gs://videoworld-storage/
Traceback (most recent call last):
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
    gsutil.RunMain()
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gsutil.py", line 123, in RunMain
    sys.exit(gslib.__main__.main())
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 429, in main
    return _RunNamedCommandAndHandleExceptions(
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions
    _HandleUnknownFailure(e)
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 625, in _RunNamedCommandAndHandleExceptions
    return command_runner.RunNamedCommand(command_name,
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
    return_code = command_inst.RunCommand()
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1190, in RunCommand
    self.Apply(_CopyFuncWrapper,
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
    self._ParallelApply(
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
    self._CreateNewConsumerPool(process_count, thread_count,
  File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
    p.start()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

The error is triggered when attempting to use -m flag and the system attempting to use Python3.8. The solution is to force the command to use Python3.7 or other compatible version.

This is accomplished by setting an environment variable either in the shell you need it in, or by adding the lines to .bashrc or .zshrc or to the startup script of your choice of shell.

$ export CLOUDSDK_PYTHON=/usr/bin/python3     # on mac
$ export CLOUDSDK_PYTHON=/usr/bin/python3.7   # on linux

Now gsutil -m will work and your command will use threading!

Subscribe to the cloud.weekly newsletter!

I write a weekly-ish newsletter on cloud, devops, and privacy called cloud.weekly!
It features the latest news, guides & tutorials and new open source projects. You can sign up via email below.