1. Error description

Routine roast , The first time I encountered this error , I am very speechless . In the past, there was no error reporting , Compared with before , Different places have

1. There's more data , from 80 Example becomes 100 example
2. A new one docker Mirror image , Probably pytorch Version and cuda There are some problems with the version
3. Code checked , No changes

2. Try yourself

2.1 Reduce batch_size

It may have something to do with video memory

For the first time

• Combine your own code , I'm the first epoch Training done ,validation The error reported in the stage （line 243）
• Adjust after reporting an error batch_size smaller （10→8）, Continue to report mistakes

• But the last step validation The place of has gone ,line 258 Report errors
• Explain batch_size Getting smaller has a certain effect

According to this idea, continue to batch_size The small （8→5）, Another position error ..

• line 305 Report errors

2.2 Change card and code

Change your mind

• Changed a card , from 0 The card has changed 1 card ..
• Deleted non essential CPU and GPU Exchange the code of the data part
• Still wrong
  
• Use two cards ,batch_size Set up 10, No pre training model , Or an error
  
• Training from scratch , Single card ,batch_size=5
  
• Training from scratch , Single card ,batch_size=4
  
  Things are a little better , All run to No 8 individual epoch 了 . But it still broke

2.3 cuda and pytorch edition

The current mirror cuda and pytorch The version information is ：

• CUDA Version: 11.6
• pytorch 1.8.0

according to pytorch file ：INSTALLING PREVIOUS VERSIONS OF PYTORCH

In terms of version , It's really not a good match .

in addition , according to TensorFlow、PyTorch Corresponding to each version CUDA、cuDNN Relationship

It's true that the version doesn't match very well .

Try changing the old mirror image of previous training

• Find the old mirror cuda Version and pytorch The version is the same as above
• That is, someone's response to the server cuda The version has been upgraded , Then upgrade the container by yourself pytorch Version .

3. Research situation

The error message is CUDA A runtime error thrown out , Illegal memory access occurred . There are also many discussions about this issue on the Internet , But no real reason was found .

Many are based on feelings

Reference resources ：

• pytorch Of github issue：RuntimeError: CUDA error: an illegal memory access was encountered
  
  • This answer seems to be effective for more people , A painful debug The experience of -RuntimeError: CUDA error: an illegal memory access was encountered, That's how this man solved it
• Others are empirical ,
  • CSDN Blog ：[ Thorough solution ]CUDA error: an illegal memory access was encountered(CUDA error Illegal access to memory )
• yolo Of GitHub issue：Cuda illegal memory access when running inference on *.engine #6311

4. My solution

It's not hard to see , I report the wrong position basically from gpu Go to cpu Problems during conversion .

• So consider whether cpu Not enough memory , So there is an error in memory access
• Because I use containers , So in docker-compose perhaps dockerfile Change the configuration item to ：

      shm_size: 64G  →     shm_size: 128G
  

• shm_size, Shared memory （shared memory）
• After that, there is basically no error report ...