error when build tensorflow 1.13.1 with CUDA 10.1

[alanpurple]

What version of protobuf and what language are you using?
Version: 3.6.1
Language: C++ build using for python

What operating system (Linux, Windows, ...) and version?
Ubuntu 18.04.2 x64

What runtime / compiler are you using (e.g., python version or gcc version)
gcc 7.3.0

What did you do?
Steps to reproduce the behavior:
initiate tensorflow build

What did you expect to see
seamlessly built

What did you see instead?
protobuf error

./tensorflow/core/framework/node_def_util.h:167:177: required from here
external/protobuf_archive/src/google/protobuf/map.h:422:29: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsNonEmptyList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
if (m->TableEntryIsNonEmptyList(bucket_index)) {
~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h:425:30: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsTree(google::protobuf::Map<Key, T>::size_type) const [with Key = std::cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
} else if (m->TableEntryIsTree(bucket_index)) {
~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h: In instantiation of 'bool google::protobuf::Map<Key, T>::InnerMap::iterator_base::revalidate_if_necessary(google::protobuf::Map<Key, T>::InnerMap::TreeIterator*) [with KeyValueType = google::protobuf::Map<std::__cxx11::basic_string, tensorflow::AttrValue>::KeyValuePair; Key = std::__cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::InnerMap::TreeIterator = std::_Rb_tree_const_iterator<std::__cxx11::basic_string>]':
external/protobuf_archive/src/google/protobuf/map.h:603:12: required from 'void google::protobuf::Map<Key, T>::InnerMap::erase(google::protobuf::Map<Key, T>::InnerMap::iterator) [with Key = std::__cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::InnerMap::iterator = google::protobuf::Map<std::__cxx11::basic_string, tensorflow::AttrValue>::InnerMap::iterator_base<google::protobuf::Map<std::__cxx11::basic_string, tensorflow::AttrValue>::KeyValuePair>]'
external/protobuf_archive/src/google/protobuf/map.h:1132:1: required from 'google::protobuf::Map<Key, T>::iterator google::protobuf::Map<Key, T>::erase(google::protobuf::Map<Key, T>::iterator) [with Key = std::__cxx11::basic_string; T = tensorflow::AttrValue]'
external/protobuf_archive/src/google/protobuf/map.h:1137:14: required from 'void google::protobuf::Map<Key, T>::erase(google::protobuf::Map<Key, T>::iterator, google::protobuf::Map<Key, T>::iterator) [with Key = std::__cxx11::basic_string; T = tensorflow::AttrValue]'
external/protobuf_archive/src/google/protobuf/map.h:1140:21: required from 'void google::protobuf::Map<Key, T>::clear() [with Key = std::__cxx11::basic_string; T = tensorflow::AttrValue]'
external/protobuf_archive/src/google/protobuf/map_field_inl.h:178:1: required from 'void google::protobuf::internal::MapField<Derived, Key, T, key_wire_type, value_wire_type, default_enum_value>::Clear() [with Derived = tensorflow::NameAttrList_AttrEntry_DoNotUse; Key = std::__cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::internal::WireFormatLite::FieldType kKeyFieldType = (google::protobuf::internal::WireFormatLite::FieldType)9; google::protobuf::internal::WireFormatLite::FieldType kValueFieldType = (google::protobuf::internal::WireFormatLite::FieldType)11; int default_enum_value = 0]'
bazel-out/k8-opt/genfiles/tensorflow/core/framework/attr_value.pb.h:1785:15: required from here
external/protobuf_archive/src/google/protobuf/map.h:484:29: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsNonEmptyList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
if (m->TableEntryIsNonEmptyList(bucket_index)) {
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h:498:24: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::cxx11::basic_string; T = tensorflow::AttrValue; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
return m->TableEntryIsList(bucket_index);
~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h: In instantiation of 'void google::protobuf::Map<Key, T>::InnerMap::iterator_base::SearchFrom(google::protobuf::Map<Key, T>::size_type) [with KeyValueType = google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::KeyValuePair; Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::Map<Key, T>::size_type = long unsigned int]':
external/protobuf_archive/src/google/protobuf/map.h:394:11: required from 'google::protobuf::Map<Key, T>::InnerMap::iterator_base::iterator_base(const google::protobuf::Map<Key, T>::InnerMap) [with KeyValueType = google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::KeyValuePair; Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map.h:510:28: required from 'google::protobuf::Map<Key, T>::InnerMap::iterator google::protobuf::Map<Key, T>::InnerMap::begin() [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::Map<Key, T>::InnerMap::iterator = google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::InnerMap::iterator_base<google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::KeyValuePair>]'
external/protobuf_archive/src/google/protobuf/map.h:1030:27: required from 'google::protobuf::Map<Key, T>::iterator google::protobuf::Map<Key, T>::begin() [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map.h:1140:27: required from 'void google::protobuf::Map<Key, T>::clear() [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map_field_inl.h:178:1: required from 'void google::protobuf::internal::MapField<Derived, Key, T, key_wire_type, value_wire_type, default_enum_value>::Clear() [with Derived = tensorflow::FunctionDef_RetEntry_DoNotUse; Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::internal::WireFormatLite::FieldType kKeyFieldType = (google::protobuf::internal::WireFormatLite::FieldType)9; google::protobuf::internal::WireFormatLite::FieldType kValueFieldType = (google::protobuf::internal::WireFormatLite::FieldType)9; int default_enum_value = 0]'
bazel-out/k8-opt/genfiles/tensorflow/core/framework/function.pb.h:776:14: required from here
external/protobuf_archive/src/google/protobuf/map.h:422:29: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsNonEmptyList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::__cxx11::basic_string; T = std::cxx11::basic_string; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
if (m->TableEntryIsNonEmptyList(bucket_index)) {
~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h:425:30: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsTree(google::protobuf::Map<Key, T>::size_type) const [with Key = std::__cxx11::basic_string; T = std::cxx11::basic_string; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
} else if (m->TableEntryIsTree(bucket_index)) {
~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h: In instantiation of 'bool google::protobuf::Map<Key, T>::InnerMap::iterator_base::revalidate_if_necessary(google::protobuf::Map<Key, T>::InnerMap::TreeIterator*) [with KeyValueType = google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::KeyValuePair; Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::Map<Key, T>::InnerMap::TreeIterator = std::_Rb_tree_const_iterator<std::__cxx11::basic_string*>]':
external/protobuf_archive/src/google/protobuf/map.h:603:12: required from 'void google::protobuf::Map<Key, T>::InnerMap::erase(google::protobuf::Map<Key, T>::InnerMap::iterator) [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::Map<Key, T>::InnerMap::iterator = google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::InnerMap::iterator_base<google::protobuf::Map<std::__cxx11::basic_string, std::__cxx11::basic_string >::KeyValuePair>]'
external/protobuf_archive/src/google/protobuf/map.h:1132:1: required from 'google::protobuf::Map<Key, T>::iterator google::protobuf::Map<Key, T>::erase(google::protobuf::Map<Key, T>::iterator) [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map.h:1137:14: required from 'void google::protobuf::Map<Key, T>::erase(google::protobuf::Map<Key, T>::iterator, google::protobuf::Map<Key, T>::iterator) [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map.h:1140:21: required from 'void google::protobuf::Map<Key, T>::clear() [with Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string]'
external/protobuf_archive/src/google/protobuf/map_field_inl.h:178:1: required from 'void google::protobuf::internal::MapField<Derived, Key, T, key_wire_type, value_wire_type, default_enum_value>::Clear() [with Derived = tensorflow::FunctionDef_RetEntry_DoNotUse; Key = std::__cxx11::basic_string; T = std::__cxx11::basic_string; google::protobuf::internal::WireFormatLite::FieldType kKeyFieldType = (google::protobuf::internal::WireFormatLite::FieldType)9; google::protobuf::internal::WireFormatLite::FieldType kValueFieldType = (google::protobuf::internal::WireFormatLite::FieldType)9; int default_enum_value = 0]'
bazel-out/k8-opt/genfiles/tensorflow/core/framework/function.pb.h:776:14: required from here
external/protobuf_archive/src/google/protobuf/map.h:484:29: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsNonEmptyList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::__cxx11::basic_string; T = std::cxx11::basic_string; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
if (m->TableEntryIsNonEmptyList(bucket_index)) {
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
external/protobuf_archive/src/google/protobuf/map.h:498:24: error: cannot call member function 'bool google::protobuf::Map<Key, T>::InnerMap::TableEntryIsList(google::protobuf::Map<Key, T>::size_type) const [with Key = std::__cxx11::basic_string; T = std::cxx11::basic_string; google::protobuf::Map<Key, T>::size_type = long unsigned int]' without object
return m->TableEntryIsList(bucket_index);
~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
ERROR: /home/wproject/repo/tensorflow/tensorflow/core/kernels/BUILD:3795:1: output 'tensorflow/core/kernels/_objs/topk_op_gpu/topk_op_gpu.cu.pic.o' was not created
ERROR: /home/wproject/repo/tensorflow/tensorflow/core/kernels/BUILD:3795:1: not all outputs were created or valid

Anything else we should know about your project / environment

tensorflow/tensorflow#26155

[acozzette]

@alanpurple That is a pretty mysterious error and I can't figure out what it is. If it's appearing for certain GCC versions but not others then that does make me wonder if we're hitting a compiler bug.

[artur-braga]

Is this problem further investigated ? I'm having unfortunately the same error

[edrozenberg]

Same problem here. Some related posts suggest downgrading from cuda 10.1 to cuda 10.0 but when I try to do that the cuda 10.0 installer fails saying it does not support my compiler gcc 8.3. Pretty much stuck atm due to this issue.

[edrozenberg]

Please see tensorflow/tensorflow#26155 (comment) for a workaround. The workaround involves patching the protobuf source code file map.h and modifying the tensorflow source code workspace configuration to utilize this patch.

[rongou]

This should be fixed with CUDA 10.1 Update 1.

[alanpurple]

fixed by cuda update

[LightingMc]

I am facing a similar error. How do I get that particular update for CUDA? @alanpurple @rongou?