Another huge batch #1751

AndreaLanfranchi · 2018-12-05T17:07:11Z

A lot of rework

This PR is the result of voluntary rework on many parts of the code to deduplicate as much as possible and give common functions some more usability

Api Interface

I've managed to unify json socket api interface with http interface. This helps to keep only one listening endpoint and removes the dependency from MONGOOSE external project.
The implementation can be now easily extended to support all API methods with GET and POST http verbs. (original idea from #1163)

Due to the recently inserted device abstraction we're now able to expose PCI id in both API calls and Http stats output along with the mode the device has been subscribed (cuda/openCL). (see #1012)

Note in this sample the power drain is not displayed as Nvidia Gtx 1050 Ti does not allow this feature but for other GPUs the power drain is reported correctly if proper -HWMON value is set.

To get the Http info page you have to point to the very same endpoint of your API interface. The page auto-refreshes every 10 seconds.

The API call miner_getstatHr have been dropped (it was only ethminer's) and the new and extended miner_getstatdetail now produces this output

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "connection": {                                     // Current active connection
      "connected": true,
      "switches": 1,
      "uri": "stratum1+tls12://<ethaddress>[email protected]:5555"
    },
    "devices": [                                        // Array subscribed of devices
      {
        "_index": 0,                                    // Miner ordinal 
        "_mode": "CUDA",                                // Miner mode : "OpenCL" / "CUDA"
        "hardware": {                                   // Device hardware info
          "name": "GeForce GTX 1050 Ti 3.95 GB",        // Name
          "pci": "01:00.0",                             // Pci Id
          "sensors": [                                  // An array made of ...
            47,                                         //  + Detected temp
            70,                                         //  + Fan percent
            0                                           //  + Power drain in watts
          ],
          "type": "GPU"                                 // Device Type : "CPU" / "GPU" / "ACCELERATOR"
        },
        "mining": {                                     // Mining info
          "hashrate": "0x0000000000e3fcbb",             // Current hashrate in hashes per second
          "pause_reason": null,                         // If the device is paused this contains the reason
          "paused": false,                              // Wheter or not the device is paused
          "segment": [                                  // The search segment of the device
            "0xbcf0a663bfe75dab",                       //  + Lower bound
            "0xbcf0a664bfe75dab"                        //  + Upper bound
          ],
          "shares": [                                   // Shares / Solutions stats
            1,                                          //  + Found shares
            0,                                          //  + Rejected (by pool) shares
            0,                                          //  + Failed shares (always 0 if --no-eval is set)
            15                                          //  + Time in seconds since last found share
          ]
        }
      },
      { ... }                                           // Another device
      { ... }                                           // And another ...
    ],
    "host": {
      "name": "miner01",                                // Host name of the computer running ethminer
      "runtime": 121,                                   // Duration time (in seconds)
      "version": "ethminer-0.18.0-alpha.1+commit.70c7cdbe.dirty"
    },
    "mining": {                                         // Mining info for the whole instance
      "difficulty": 3999938964,                         // Actual difficulty in hashes
      "epoch": 227,                                     // Current epoch
      "epoch_changes": 1,                               // How many epoch changes occurred during the run
      "hashrate": "0x00000000054a89c8",                 // Overall hashrate (sum of hashrate of all devices)
      "shares": [                                       // Shares / Solutions stats
        2,                                              //  + Found shares
        0,                                              //  + Rejected (by pool) shares
        0,                                              //  + Failed shares (always 0 if --no-eval is set)
        15                                              //  + Time in seconds since last found share
      ]
    },
    "monitors": {                                       // A nullable object which may contain some triggers
      "temperatures": [                                 // Monitor temperature
        60,                                             //  + Resume mining if device temp is <= this threshold
        75                                              //  + Suspend mining if device temp is >= this threshold
      ]
    }
  }
}

This PR contains also integration to documentation.

Solution submission

I've discovered that each miner having it's own strand in it's own thread could have lead to collisions while submitting. I managed to have each miner submit to Farm's strand so the solution submission to socket is guaranteed to be serialized with the benefit of an overhead reduction due to only one relevant strand instead of one per miner. (less work for the io_service).

Solution and stats data accounting

Until now all accounting operations were somewhat disorganized (technically there was one vector for each type of information to be collected (solutions and their conditions, hashrates, status of miners) which also implied there were several functions to call to pull data from farm. In addition all vectors where increased dynamically with a lot of checks like if (vector.size() < index).
I've reworked all accounting under a single structure TelemetryType which holds several members of TelemetryAccountType which is initialized with room for each miner thread.
Each structure type has it's own str() method which helps to remove a lot of definition for operator << while allowing to treat the output as strings.

Terminal logging

The output of the rework appears like this (with LOG_PER_GPU) enabled

 m 17:49:46 ethminer 0:25 A30 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 46C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.58 56C 70% A5 | cu5 14.74 55C 70% A7 }
 m 17:49:51 ethminer 0:25 A30 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 45C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A7 }
 i 17:49:53 ethminer Job: #34597da8… block 6831664 eu1.ethermine.org  [172.65.207.106:5555]
 m 17:49:56 ethminer 0:25 A30 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 46C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A7 }
 m 17:50:01 ethminer 0:26 A30 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 45C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A7 }
 m 17:50:06 ethminer 0:26 A30 88.21 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 46C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A7 }
 m 17:50:11 ethminer 0:26 A30 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 46C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A7 }
cu 17:50:13 cuda-5   Job: #34597da8… Sol: 0x4a2728c28f4ddd83
 i 17:50:13 ethminer **Accepted  53 ms. eu1.ethermine.org  [172.65.207.106:5555]
 m 17:50:16 ethminer 0:26 A31 88.22 Mh { cu0 14.75 48C 70% A9 | cu1 14.75 45C 70% A0 | cu2 14.74 51C 70% A4 | cu3 14.66 55C 70% A5 | cu4 14.57 56C 70% A5 | cu5 14.74 55C 70% A8 }

The first 3 chunks of the log line are

Run duration
Overall solutions detail (same format Ax:Wx:Rx:Fx) where A means accepted, W means wasted (solutions produced when there is no connection available - eg. during a switch), R means rejected, F means failed (only if --no-eval is not set)
Actual hashing speed

In curly braces (array) the detail for each miner which is now labelled by a more significant cu/cl prefix (cuda or opencl) instead of the previous generic "gpu"
Each miner reports it's hashing speed (with the same order of magnitude of the overall hashing speed -ie if the farm speed is in Mh then all miner's speeds are in Mh too), the sensors values (if --HWMON) and the detail of their solutions (if LOG_PER_GPU) : this latter removes the need for a double line printout. (see #1605)

Also added documentation

When solution is found also output the job it refers.

List of devices is a space separated list.

Also enforce usage of diff in genesis block for benchmark

Simulation and benchmark are de-facto the same thing. User can test hashing speed using simulation. It's assumed difficulty 1 (roughly 4.3 GH/s) Note ! Data exposed by benchmark (now removed) were also wrong as the "min" and "max" values exposed were in reality the first and the last collected value.

This reverts commit 885d8e0.

Allocating the (current ~3GB) DAG needs 5 sec in my environment. This patch reuses the already allocated memory if possible.

* some optimization using lop3 * use ROL8/ROR8 for some cases

This reverts commit 6ecadec.

This reverts commit b1167b1.

Use single strand from farm instead of having multiple strands (one per miner) which may collide)

A wasted solution is a solution found which could not be sent to pool due to lack of connection.

Compute max and mean. Tidy log.

StefanOberhumer · 2018-12-05T21:59:46Z

libpoolprotocols/PoolManager.cpp

 {
-    Guard l(m_activeConnectionMutex);


Why removed ? Preventing another API call modifying connections the same time or switching to next failoverpool while we change connections

I pushed a test commit nevertheless all mutexes are not necessary as the API lives in the same thread. There is only one thread for all which is not a miner

In other words ethminer threads are (1 + device number)

In theory (correct me if I'm wrong)

there could be more than one API connection

API could "half" added the connection (means: size of URI vectors increased but URI content not copied) while poolmanager tries to use next (not fully filled) connection

All actions from any number of API connections are executed serialized by io_service.
There is no concurrency to worry about. There's to worry about asynchrony. What can happen is that a call (eg. Setactiveconnection) does not complete it's whole cycle before another call kicks in

StefanOberhumer · 2018-12-05T22:01:06Z

libpoolprotocols/PoolManager.cpp

@@ -247,10 +247,11 @@ void PoolManager::stop()
    DEV_BUILD_LOG_PROGRAMFLOW(cnote, "PoolManager::stop() end");
 }

-void PoolManager::addConnection(URI& conn)
+void PoolManager::addConnection(URI& _conn)


Wouldn't it be enough to change function to void PoolManager::addConnection(URI conn) { (just removed reference) ?

StefanOberhumer · 2018-12-05T23:37:55Z

Could we clang-format (now all sources - see #1712 ) before merge ?

StefanOberhumer · 2018-12-05T23:49:18Z

Counting solutions:
Not sure but I think with this commit we count per miner and also the totals of (eg rejected) solutions.
As the increments are not guarded it could be possible that the sum of all miners will not match the total sum.
I think in current version I tried to compute the totals "on the fly" by summing all miner stats to prevent such a difference and tried not using a lock.

AndreaLanfranchi · 2018-12-06T00:15:14Z

Again. No need to guard increments as they come from a single serialized strand.

ghost · 2018-12-06T05:02:57Z

Can you rebase your commits with master branch?

AndreaLanfranchi · 2018-12-06T08:40:39Z

@naikmyeong this is master branch

StefanOberhumer

As consequence I'll remove adapted documentation from #1743

AndreaLanfranchi and others added 30 commits November 29, 2018 12:00

Remove unused oclPlatform filter from CLI

daa3520

Help text for new URI parsing mode.

a0b1cb8

Uri parsing allowing BOTH backticks enclosures or url_encoding

a10abce

Also added documentation

Minor log fix

fc8e38a

When solution is found also output the job it refers.

Log format consistency

8d82c9e

Amend wrong help text

23dc4bf

List of devices is a space separated list.

getTargetFromDiff as common function

2f428e3

Also enforce usage of diff in genesis block for benchmark

getTargetFromDiff as common function

45d666a

Also enforce usage of diff in genesis block for benchmark

Remove unused class Notified

58a3676

Avoid cpp warning

885d8e0

Revert "Avoid cpp warning"

db0d097

This reverts commit 885d8e0.

Explicit braces to avoid ambiguous else

0148dde

Removed unused classes BlockHeader and RLP

68330cc

Remove unused exceptions

d626f26

Ethminer supports NVIDIA CUDA 8.0 build

37424c0

CUDA: Only allocate DAG memory if needed.

62165dc

Allocating the (current ~3GB) DAG needs 5 sec in my environment. This patch reuses the already allocated memory if possible.

Add Cuda8 Support, Macro Name to __SHFL

ccea6b1

change MACRO name __SHFL to SHFL

bd5f8ce

Clean up CUDA-8 shuffle patch

7f37e29

Epoch and difficulty on same log line

3d7deb8

Mining gets paused only on disconnection (which is async)

fde2a01

Amend segfault error when stopping before any work got in

da5cbea

inline asm optimizations

6ecadec

* some optimization using lop3 * use ROL8/ROR8 for some cases

Some more log clean up

6e295e4

Revert "inline asm optimizations"

b1167b1

This reverts commit 6ecadec.

Revert "Revert "inline asm optimizations""

cf6dc5a

This reverts commit b1167b1.

Use HexPrefix enum option when needed

9ef9856

Move common functions to CommonData

26c3b47

Async solution submission in a single strand

f3268a2

Use single strand from farm instead of having multiple strands (one per miner) which may collide)

AndreaLanfranchi and others added 12 commits December 5, 2018 14:31

Further reduce wording in log lines

3c90053

use uint2 const to reduce vectorize()/mov.b64

29835ad

Improved readability

098ca45

Account also wasted solutions

254f0bf

A wasted solution is a solution found which could not be sent to pool due to lack of connection.

Simulation rework

0a758f3

Compute max and mean. Tidy log.

Remove port number redundancy in host log line

c52925e

Html output small fix

0004b47

Merge branch 'master' into master

434ef85

Always display block if there is available

43ea073

Remove # in front of job

3a685a8

Remove duplicated code

f560a4f

Amend travis and circle errors/warnings

4b558b5

AndreaLanfranchi requested review from StefanOberhumer, chfast, MariusVanDerWijden and smurfy December 5, 2018 18:09

AndreaLanfranchi added 2 commits December 5, 2018 21:09

Remove check for mixed stratum/getwork in API

ccaae7f

Simplify hashrate submission

419a515

StefanOberhumer reviewed Dec 5, 2018

View reviewed changes

StefanOberhumer approved these changes Dec 6, 2018

View reviewed changes

MariusVanDerWijden merged commit 6e8252f into ethereum-mining:master Dec 6, 2018

This was referenced Dec 6, 2018

Feature request: more concise and descriptive output #1605

Closed

BusId in API #1012

Closed

Variables rework #1769

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Another huge batch #1751

Another huge batch #1751

AndreaLanfranchi commented Dec 5, 2018 •

edited

Loading

StefanOberhumer Dec 5, 2018 •

edited

Loading

AndreaLanfranchi Dec 5, 2018

AndreaLanfranchi Dec 5, 2018

StefanOberhumer Dec 5, 2018 •

edited

Loading

AndreaLanfranchi Dec 5, 2018

StefanOberhumer Dec 5, 2018 •

edited

Loading

StefanOberhumer commented Dec 5, 2018

StefanOberhumer commented Dec 5, 2018 •

edited

Loading

AndreaLanfranchi commented Dec 6, 2018

ghost commented Dec 6, 2018

AndreaLanfranchi commented Dec 6, 2018

StefanOberhumer left a comment

Another huge batch #1751

Another huge batch #1751

Conversation

AndreaLanfranchi commented Dec 5, 2018 • edited Loading

A lot of rework

Api Interface

Solution submission

Solution and stats data accounting

Terminal logging

StefanOberhumer Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

AndreaLanfranchi Dec 5, 2018

Choose a reason for hiding this comment

AndreaLanfranchi Dec 5, 2018

Choose a reason for hiding this comment

StefanOberhumer Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

AndreaLanfranchi Dec 5, 2018

Choose a reason for hiding this comment

StefanOberhumer Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

StefanOberhumer commented Dec 5, 2018

StefanOberhumer commented Dec 5, 2018 • edited Loading

AndreaLanfranchi commented Dec 6, 2018

ghost commented Dec 6, 2018

AndreaLanfranchi commented Dec 6, 2018

StefanOberhumer left a comment

Choose a reason for hiding this comment

AndreaLanfranchi commented Dec 5, 2018 •

edited

Loading

StefanOberhumer Dec 5, 2018 •

edited

Loading

StefanOberhumer Dec 5, 2018 •

edited

Loading

StefanOberhumer Dec 5, 2018 •

edited

Loading

StefanOberhumer commented Dec 5, 2018 •

edited

Loading