Containers and Kubernetes

Get Involved. Join the Conversation.

Topic

    Olivier Maurice
    Security problem on FSS hosted volume
    Topic posted May 29, 2019 by Olivier MauriceRed Ribbon: 250+ Points, tagged Kubernetes 
    66 Views, 3 Comments
    Title:
    Security problem on FSS hosted volume
    Summary:
    Some pods give a security problem when accessing FSS hosted exports
    Content:

    Hi,

    Not new to Kubernetes but also not an expert. The setting: a Kubernetes cluster (OKE) with the storage behind the PV and PVC residing on File Storage Service (FSS).

    When making a deployment based on Alpine, I can perfectly mount and use the volume in the pod.

    However, when switching to some more meaningful stuff, say MySQL or my latest try Prometheus, I just cannot make it fly. None of these containers can work with the export. In all cases the PV and PVC are bound.

    This is something security - related but I just can't figure it out. I have been squashing the root or all users to 1 or something in the 65K, nothing seemed to help.
    Also defined security context on pod level, to no avail. I am missing something, but it is clear I do not know what.

     

    What I have in place:

    Storageclass

    kind: StorageClass
    apiVersion: storage.k8s.io/v1beta1
    metadata:
      name: oci-fss
    provisioner: oracle.com/oci-fss
    parameters:
      mntTargetId: ocid1.mounttarget.oc1.eu_frankfurt_1.aaaa...aa
    

    PV

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: prometheus-pv
      namespace: monitoring
      labels:
        app: prometheus
    spec:
      storageClassName: oci-fss
      capacity:
        storage: 100Gi
      accessModes:
        - ReadWriteMany
      mountOptions:
        - nosuid
      persistentVolumeReclaimPolicy: Delete # Reclaim policies are defined below
      nfs:
        # Replace this with the IP of your FSS file system in OCI
        server: 10.100.0.3
        # Replace this with the Path of your FSS file system in OCI
        path: "/k8s-prometheus"
        readOnly: false
    

     

    PVC
    
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: prometheus-pvc
      namespace: monitoring
    spec:
      storageClassName: oci-fss
      accessModes:
        - ReadWriteMany
      resources:
        requests:
        # Although storage is provided here it is not used for FSS file systems
          storage: 100Gi
      selector:
        matchLabels:
          app: prometheus
    

     

    Deployment

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: prometheus-deployment
      namespace: monitoring
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: prometheus-server
        spec:
          containers:
            - name: prometheus
              image: prom/prometheus:v2.2.1
              args:
                - "--config.file=/etc/prometheus/prometheus.yml"
                - "--storage.tsdb.path=/prometheus/"
              ports:
                - containerPort: 9090
              volumeMounts:
                - name: prometheus-config-volume
                  mountPath: /etc/prometheus/
                - name: prometheus-storage-volume
                  mountPath: /prometheus/
          volumes:
            - name: prometheus-config-volume
              configMap:
                defaultMode: 420
                name: prometheus-server-conf
            - name: prometheus-storage-volume
              persistentVolumeClaim:
                claimName: prometheus-pvc
                readOnly: false
    

    Log output

    level=error ts=2019-05-29T07:17:48.980589701Z caller=main.go:582 err="Opening storage failed open DB in /prometheus/: open /prometheus/199323036: permission denied"
    
    level=info ts=2019-05-29T07:17:48.980731276Z caller=main.go:584 msg="See you next time!"
    
     
    Thanks for your ideas!
     
    Olivier
    Version:
    Kubernetes v1.11.5-3

    Comment

     

    • Mike Raab

      You might look at this, it may be related

      https://github.com/coreos/prometheus-operator/issues/830

    • Olivier Maurice

      Hi,
       

      I already bumped into this page, but some of these discussions are a little over my head... :)

      Now one thing I realized is that I start with an empty share and I assume that the software in the pod creates the directories on the share. I guess part of the problem can be found here.

      I am not using an operator at this point, only fiddling with the basic 'building blocks'. Will come back with my findings when creating the directories upfront on the share.

       

      Thanks already for the feedback,

      Olivier

    • Dario Stella

      Hi, I am having the same issue but with another software running in the pods. Have you fix it?

      Thanks in advance