前提・実現したいこと
aws lambdaで機械学習の学習済みモデルを使って、aws dynamodbに入力されるデータに対する分類問題を解かせるプログラムを作ろうと考えています。
発生している問題・エラーメッセージ
機械学習のモデルをロードする上でtensorflowをインポートする必要があり、serverless frameworkでS3 bukectを通して、lambdaの/tempに展開しようと考えています。
s3からlambdaの/tmpディレクトリに展開する際にサイズオーバーでエラーが出てしまいます。
terminal
1"errorMessage": "[Errno 28] No space left on device", 2 "errorType": "OSError", 3 "stackTrace": [
該当のソースコード
デプロイしようとしているserverless.ymlとrequirements.txtは以下のように設定しています。
serverless.yml
1service: sample-app 2 3provider: 4 name: aws 5 runtime: python3.8 6 region: ap-northeast-1 7 stage: dev 8 iamRoleStatements: 9 - Effect: "Allow" 10 Action: 11 - s3:ListBucket 12 - s3:GetObject 13 Resource: 14 - "arn:aws:s3::*" 15 16custom: 17 pythonRequirements: 18 dockerizePip: true 19 zip: true 20 slim: true 21 slimPatterns: 22 - "**/debug" 23 - "**/grpc" 24 - "**/h5py" 25 - "**/markdown" 26 - "**/numpy" 27 - "**/pkg_resources" 28 - "**/setuptools" 29 - "**/tensorboard/plugins" 30 - "**/tensorboard/webfiles.zip" 31 - "**/tensorflow_core/contrib" 32 - "**/tensorflow_core/examples" 33 - "**/tensorflow_core/include" 34 - "**/tensorflow_estimator" 35 - "**/werkzeug" 36 - "**/wheel" 37 38 39functions: 40 hello: 41 handler: handler.hello 42 timeout: 60 43 44plugins: 45 - serverless-python-requirements
requirements.txt
1tensorflow==2.2.0rc3
試したこと
・tensorflowのバージョンを変更した。
tensorflow2.2.0rc0をrequireする
→tensorflow2.2.0rc1以上のバージョンを使えというエラー?
error
1STDERR: ERROR: Could not find a version that satisfies the requirement tensorflow==2.2.0rc0 (from -r /var/task/requirements.txt (line 1)) (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1, 2.4.0rc2, 2.4.0rc3)
tensorflow2.2.0rc1をrequireする
→
error
1Error: STDOUT: Collecting tensorflow==2.2.0rc1 2 Using cached tensorflow-2.2.0rc1-cp38-cp38-manylinux2010_x86_64.whl (516.2 MB) 3 Collecting six>=1.12.0 4 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) 5 Collecting wheel>=0.26; python_version >= "3" 6 Using cached wheel-0.36.0-py2.py3-none-any.whl (34 kB) 7 Collecting keras-preprocessing>=1.1.0 8 Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB) 9 Collecting grpcio>=1.8.6 10 Downloading grpcio-1.34.0-cp38-cp38-manylinux2014_x86_64.whl (4.0 MB) 11 Collecting google-pasta>=0.1.8 12 Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB) 13 Collecting h5py<2.11.0,>=2.10.0 14 Using cached h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB) 15 Collecting numpy<2.0,>=1.16.0 16 Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB) 17 Collecting gast==0.3.3 18 Using cached gast-0.3.3-py2.py3-none-any.whl (9.7 kB) 19 Collecting opt-einsum>=2.3.2 20 Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB) 21 Collecting absl-py>=0.7.0 22 Using cached absl_py-0.11.0-py3-none-any.whl (127 kB) 23 Collecting tensorflow-estimator<2.3.0,>=2.2.0rc0 24 Using cached tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB) 25 Processing /var/useDownloadCache/wheels/5f/fd/9e/b6cf5890494cb8ef0b5eaff72e5d55a70fb56316007d6dfe73/wrapt-1.12.1-cp38-cp38-linux_x86_64.whl 26 Collecting scipy==1.4.1; python_version >= "3" 27 Using cached scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB) 28 Collecting protobuf>=3.8.0 29 Using cached protobuf-3.14.0-cp38-cp38-manylinux1_x86_64.whl (1.0 MB) 30 Collecting tensorboard<2.2.0,>=2.1.0 31 Using cached tensorboard-2.1.1-py3-none-any.whl (3.8 MB) 32 Collecting astunparse==1.6.3 33 Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB) 34 Processing /var/useDownloadCache/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501/termcolor-1.1.0-py3-none-any.whl 35 Collecting requests<3,>=2.21.0 36 Using cached requests-2.25.0-py2.py3-none-any.whl (61 kB) 37 Collecting google-auth<2,>=1.6.3 38 Using cached google_auth-1.23.0-py2.py3-none-any.whl (114 kB) 39 Collecting markdown>=2.6.8 40 Using cached Markdown-3.3.3-py3-none-any.whl (96 kB) 41 Collecting werkzeug>=0.11.15 42 Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB) 43 Collecting setuptools>=41.0.0 44 Using cached setuptools-50.3.2-py3-none-any.whl (785 kB) 45 Collecting google-auth-oauthlib<0.5,>=0.4.1 46 Using cached google_auth_oauthlib-0.4.2-py2.py3-none-any.whl (18 kB) 47 Collecting certifi>=2017.4.17 48 Using cached certifi-2020.11.8-py2.py3-none-any.whl (155 kB) 49 Collecting chardet<4,>=3.0.2 50 Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB) 51 Collecting urllib3<1.27,>=1.21.1 52 Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB) 53 Collecting idna<3,>=2.5 54 Using cached idna-2.10-py2.py3-none-any.whl (58 kB) 55 Collecting pyasn1-modules>=0.2.1 56 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) 57 Collecting rsa<5,>=3.1.4; python_version >= "3.5" 58 Using cached rsa-4.6-py3-none-any.whl (47 kB) 59 Collecting cachetools<5.0,>=2.0.0 60 Using cached cachetools-4.1.1-py3-none-any.whl (10 kB) 61 Collecting requests-oauthlib>=0.7.0 62 Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB) 63 Collecting pyasn1<0.5.0,>=0.4.6 64 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) 65 Collecting oauthlib>=3.0.0 66 Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB) 67 Installing collected packages: six, wheel, numpy, keras-preprocessing, grpcio, google-pasta, h5py, gast, opt-einsum, absl-py, tensorflow-estimator, wrapt, scipy, protobuf, certifi, chardet, urllib3, idna, requests, setuptools, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, markdown, werkzeug, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, astunparse, termcolor, tensorflow 68 Successfully installed absl-py-0.11.0 astunparse-1.6.3 cachetools-4.1.1 certifi-2020.11.8 chardet-3.0.4 gast-0.3.3 google-auth-1.23.0 google-auth-oauthlib-0.4.2 google-pasta-0.2.0 grpcio-1.34.0 h5py-2.10.0 idna-2.10 keras-preprocessing-1.1.2 markdown-3.3.3 numpy-1.19.4 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.25.0 requests-oauthlib-1.3.0 rsa-4.6 scipy-1.4.1 setuptools-50.3.2 six-1.15.0 tensorboard-2.1.1 tensorflow-2.2.0rc1 tensorflow-estimator-2.2.0 termcolor-1.1.0 urllib3-1.26.2 werkzeug-1.0.1 wheel-0.36.0 wrapt-1.12.1
・kerasのみをrequireした
tensorflowが必要というエラー
補足情報(FW/ツールのバージョンなど)
○環境はこのようになっています。
Your Environment Information ---------------------------
Operating System: darwin
Node Version: 12.18.1
Framework Version: 2.13.0 (standalone)
Plugin Version: 4.1.2
SDK Version: 2.3.2
Components Version: 3.4.2
○参考にしたサイト
AWS LambdaでTensorFlow 2.0を使った画像分類
バッドをするには、ログインかつ
こちらの条件を満たす必要があります。